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1.1 


Introduction 


The U.S. PIRLS and ePIRLS 2016 Technical Report and User's Guide provides an 
overview of the design and implementation of the Progress in International Reading 
Literacy Study (PIRLS) 2016 in the United States, along with information designed 
to facilitate access to the U.S. PIRLS 2016 data. 


This section provides an overview of the study and its components. Subsequent 
chapters provide details of the sampling design (chapter 2), response rates and 
nonresponse bias (chapter 3), data collection operations (chapter 4), and the data 
files (chapter 5). 


Background and Purpose of PIRLS 2016 


PIRLS is an international comparative study of student performance in reading 
literacy at the fourth grade. PIRLS 2016 marks the fourth iteration of the study, 
which has been conducted every 5 years since 2001. New to the PIRLS assessment in 
2016, ePIRLS provides a computer-based extension to PIRLS, assessing students’ 
comprehension of online information. A subset of the participating PIRLS 2016 
education systems participated in ePIRLS. 


Internationally, the study was developed and implemented by the International 
Association for the Evaluation of Educational Achievement (IEA) and is designed to 
measure the reading knowledge and skills of fourth-grade students over time. Each 
country or education system is responsible for collecting its own data following 
detailed international requirements for target populations, sampling design, sampling 
size, exclusions, assessment administration, and defining participation rates. In the 
United States, PIRLS is conducted by the National Center for Education Statistics 
(NCES), part of the Institute of Education Sciences (IES) in the U.S. Department of 
Education. NCES contracted RTI to conduct the sampling and data collection 
activities for PIRLS and ePIRLS 2016. 


The PIRLS assessment is designed to broadly align with the curricula of participating 
countries and thereby assess concepts that are presumably taught in most fourth- 
grade classrooms. In addition to the reading assessment, students provide 
background information. School administrators and reading teachers also provide 


contextual information about education, school, and classroom policies, which 
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1.2 


allows for cross-national comparisons of educational contexts that may be related to 


student achievement. 


PIRLS was first administered to students in 36 education systems in 2001." In 2016, 
58 education systems’ participated in the PIRLS assessment at the fourth grade, with 
16 of them also participating in ePIRLS. PIRLS (and its partner assessment ePIRLS) 
targets students as they have transitioned from a focus on learning to read to a focus 
on reading to learn. In most education systems, this point is the fourth year of 
formal schooling or fourth grade, with an average student age of 9.5 years. For ease 
of presentation, student participants are referred to as fourth-grade students in 


reports and findings. 


A detailed description of PIRLS 2016 from an international perspective can be found 
in reports published by the IEA and available online at 


https://timssandpirls.bc.edu/pirls2016/index.html. 


PIRLS 2016 U.S. Data Collection Activities and Schedule 


Descriptions of data collection activities and their timing within the United States 
provide a foundation for researchers seeking to understand the data. These activities 
are summarized in exhibit 1-1 along with the timing of their implementation. The 


activities are described in detail in chapters 2 through 4 of this report. 


! The term “education system” refers to IEA member countries and benchmarking participants. IEA 
member “countries” may be complete, independent political entities or nonnational entities that 
represent a portion of a member country (e.g., England, Hong Kong). Nonnational entities that are 
represented by their larger country in the main results (e.g., Moscow City in Russia, Abu Dhabi in the 
United Arab Emirates, Ontario in Canada), or whose countries are not IEA members (Buenos Aires), 
are designated as “benchmarking participants.” 

? This count differs from the totals in the international results because it excludes those education 
systems that administered the assessments to students in grades other than grade 4. PIRLS 2016 was 
administered in a total of 61 education systems. 
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Exhibit 1-1. Schedule for U.S. PIRLS 2016 data collection activities 


2014 2015 2016 


Activity Nov | Dec | Jan | Feb | Mar | Apr |May| Jun | Jul | Aug| Sep | Oct | Nov | Dec | Jan | Feb | Mar | Apr | May | Jun 


Jul 


Aug 


OMB approval received for PIRLS 2016 


School sampling 


State, district, and school recruitment a Na a 


Instrumentation 


Cultural adaptation made to items 


Production of Instruments 


Printing of instruments 


Preparing ePIRLS USB sticks 


Test administrator training 


Within-school sampling 


Class Listing Forms 
Student-Teacher Linkage Forms 


Student Tracking Forms 


Assessment sessions in schools (PIRLS) 


Assessment sessions in schools (ePIRLS) 


Data processing 


Data receipt 

Scoring and coding 

Data entry 

Cleaning and preparation of data files 


Submission of data files to the 
Data Processing Center 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


1.3. Overview of the Design and Administration of PIRLS 
and ePIRLS 2016 


The basic parameters of the design and administration of PIRLS 2016 in the United 
States are outlined below. A more detailed treatment is provided in subsequent 
chapters of this report. Methods and Procedures in PIRLS 2016 (Martin, Mullis, and 
Hooper 2017) details study design and coordination activities from an international 


perspective. 


1.3.1 Sampling 


In the United States and most other education systems, the target population of 
students corresponded to students in grade 4. The United States used a two-stage 
stratified sampling design to sample the target population, with the first stage being 
the selection of schools and the second stage being the selection of classrooms 


within schools. 


Schools were selected with a probability proportional to the size of their fourth- 
grade enrollment and from three stratification variables to make up a nationally 
representative sample. For each sampled school, two replacement schools were 


selected. 
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Intact® fourth-grade classrooms were selected within each participating school. Small 
classrooms were combined into what were called pseudo-classtooms so that each 
classroom in the sampling frame had at least 20 students. One to two classrooms 
(including pseudo-classrooms) were selected from each school by the within-school 
sampling software provided by the IEA’s Data Processing Center (DPC). All 
students enrolled in the sampled classrooms and pseudo-classrooms were selected to 
participate in the study. The teacher of each selected class and the school 


administrator were also selected to complete a survey. 


The PIRLS 2016 sample included 176 public and private schools. Replacement 
schools for each of the 176 selected schools were contacted about the study only if 


the original school refused to participate. All schools, classrooms, and students 
selected for PIRLS were also selected for ePIRLS. 


More detail is provided in chapter 2. 


1.3.2 Assessment Design 


PIRLS. The assessment instruments included fourth-grade-level stories and 
informational texts collected from several different countries. Students are asked to 
engage in a full repertoire of reading skills and strategies, including retrieving and 
focusing on specific ideas, making simple and more complex inferences, and 
examining and evaluating text features. The passages were followed by open-ended 


and multiple-choice questions about the text. 


The 2016 assessment consisted of 15 booklets and one reader.* The assessment was 
given in two 40-minute parts with a 5- to 10-minute break in between. Each of the 
booklets contained two parts—one block containing a literary experience passage 
and associated test items, and one block containing an informational passage and 
items—and each block occurred twice across the 15 total booklets. Each student 
received one booklet as part of the assessment. The entire assessment consisted of 
12 blocks of passages and items, and using different booklets allowed PIRLS to 
report results from more assessment items than can fit in one booklet, without 
making the assessment longer. To provide good coverage of each skill domain, the 
test items developed required about 8 hours of testing time. However, testing time 


was limited to 80 minutes per student by clustering items in blocks and randomly 


3 Intact or whole classes were selected from each school and all students enrolled in the selected 
classrooms were selected to participate in PIRLS and ePIRLS. 

‘The reader was the term designated to the 16" booklet which was different than the others in both 
presentation and in how the students provided responses. The reader was presented in a magazine- 
type format with the questions in a separate booklet. The other booklets included the passages and 
the questions in the same booklet. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


1.3.3 


1. INTRODUCTION 


rotating the blocks of items throughout the student test booklets (assessment 
booklets were randomly assigned to students within each sampled class). As a result, 
no student received all items (there were a total of 175 items on the 2016 
assessment), but each item was answered by a representative sample of students. 
This is consistent with other large-scale assessments, such as the National 
Assessment of Educational Progress (NAEP). 


A total of 12 reading passages were included in the 2016 assessment booklets used in 
all participating education systems. The passages comprised two that were used in 
PIRLS 2001, 2006, and 2011, two from 2006 and 2011, and six new passages. The 
use of common passages from the 2001 through the 2016 assessments allowed for 
the analysis of change in reading literacy over the 15-year period between 
administrations for countries that participated in these cycles. The passages, as well 
as all other study materials, were translated into the primary language or languages of 


instruction in each education system. 


ePIRLS. The ePIRLS assessment was based on the PIRLS 2016 Assessment 
Framework with a focus on student performance in online reading. This computer- 
based assessment was designed to understand how students acquite and use 
information on the internet. The assessment used a simulated Internet environment 
which allowed students to navigate through web pages to respond to questions about 
the online information. Students are guided through the web pages and questions by 


an on-screen avatar. 


Five tasks were included in the 2016 ePIRLS assessment. Each student was asked to 
complete two ePIRLS tasks, each taking about 40 minutes. The tasks are on topics 
related to science and social studies. Because this is the first administration of 
ePIRLS, all five tasks and its 91 items are new to the study. Two of the tasks are 
released to the public on the IEA website 
(http://timssandpirls.bc.edu/pirls2016/international-results /epirls /take-the-epirls- 
assessment/). 


Test Administration 


Test administration for PIRLS in the United States occurred February 16 through 
May 25, 2016. All students selected for PIRLS were also asked to complete ePIRLS, 
typically one day after the PIRLS administration. Students who missed the PIRLS 
session could take PIRLS while their counterparts completed ePIRLS, as no student 
could participate in ePIRLS without participating in PIRLS first. The administration 
was carried out by professional staff trained according to the international guidelines. 


School personnel were asked only to assist with listings of students, the identification 
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of space for testing in the school, and the specification of any parental consent 
procedures required. The Trends in International Math and Science Survey (TIMSS) 
& PIRLS International Study Center at Boston College monitored compliance with 


the standardized procedures. 


Scoring 


The PIRLS assessment items included both multiple-choice and constructed- 
response items. A scoring rubric (guide) was provided for every constructed- 
response item, and scoring procedures were specified by scoring manuals provided 
by the International Study Center. The national research coordinator (NRC) in each 
country was responsible for the scoring and coding of data in that country, following 
established guidelines. NRCs and, as appropriate, contractor staff attended scoring 
trainings conducted by the International Study Center and subsequently recruited 


and trained professional scorers to score constructed-response items. 


Scaling 


Total scores for reading in PIRLS, along with scores that reflect performance in 
specific subdomains, were estimated using item response theory (IRT) models. 
PIRLS 2016 had a scale for overall reading, subscales for the two purposes of 
reading, and two subscales for the processes of reading (combining the four 
processes into two subscales). Benchmark scores were also derived. IRT estimation 
procedures were also used to place scores from the four PIRLS assessments 
conducted in 2001, 2006, 2011, and 2016 on the same scale (the scale of the 2001 
administration). This allowed for the calculation of trends in achievement even 
though the makeup of the countries participating in PIRLS changed over time. 
ePIRLS is reported on the same scale as PIRLS, however ePIRLS differs in that it 
does not have a literary scale. Details are provided in Methods and Procedures in PIRLS 
2016 (Martin, Mullis, and Hooper 2017). The scale scores assigned to each student 
were estimated using a procedure described in section 1.3.6, with input from the IRT 


results. 


Plausible Values 


The matrix sampling approach used in the PIRLS assessments meant that no student 
responded to all the items. To accommodate the missing data generated by this 
design, during the scaling process, plausible values were estimated to characterize 
students participating in the assessment. Plausible values are imputed values and not 
test scores for individuals in the usual sense. They represent what the true 


performance of an individual might have been, had the student provided an answer. 
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They are estimated as random draws (usually five) from an empirically derived 
distribution of score values based on both the student’s observed responses to 
assessment items and on the student’s background variables. A more technical 
explanation can be found in Mesbods and Procedures in PIRLS 2016 (Martin, Mullis, and 
Hooper 2017). From the point of view of analysis, this means that each analysis must 


be repeated five times, once for each plausible value, and the results averaged. 


1.3.7 Weighting 


Responses from the groups of students were assigned sampling weights to adjust for 
the complex sample design that resulted in students having an unequal, but known, 
probability of selection. Additionally, an adjustment for school and student 
nonresponse was built into the weighting. The estimation of sampling weights was 
carried out by Statistics Canada. A detailed description is provided in Methods and 
Procedures in PIRLS 2016 (Martin, Mullis, and Hooper 2017). In analyses of the 
PIRLS data it is necessary to use sampling weights to obtain accurate population 


estimates. 


1.4 Reporting PIRLS and ePIRLS Results 


Achievement results from PIRLS are reported on a scale from 0 to 1,000, with a 
scale average of 500 and standard deviation of 100. Even though the education 
systems participating in PIRLS have changed across the assessment rounds from the 
first administration in 2001, comparisons between the 2016 results and prior results 
are still possible because the achievement scores in each of the assessments ate 
placed on a scale that is not dependent on the list of participating education systems 
in any particular year. The ePIRLS results are reported on the same scale as the 
PIRLS reading achievement results. A more detailed explanation of the assessment’s 
equating and scaling can be found in Methods and Procedures in PIRLS 2016 (Martin, 
Mullis, and Hooper 2017). 


In addition to numerical scale results, PIRLS also includes international benchmarks. 
The international benchmarks provide a way to interpret the scale scores and to 
understand how students’ proficiency in a subject varies along the assessment scale. 
The benchmarks for PIRLS describe four levels of student reading achievement, 
based on the kinds of skills and knowledge students at each score cut point would 
need to successfully answer the items. More information on the development of the 
benchmarks and the procedures used to set the score cut points can be found in 
Methods and Procedures in PIRLS 2016 (Martin, Mullis, and Hooper 2017). 
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1.5 U.S. International and National Data Files 


PIRLS 2016 data are available in three different formats: 


e The PIRLS U.S. international data files are available as part of the 
international database released by the International Study Center. The U.S.- 
ise PIRLS data files can be downloaded from 


These data files conform to the international specifications common to the 
data files from all countries. However, they do not include the U.S.-specific 
adaptations made to a few questions in the questionnaires or the additional 
questions added to the school and student questionnaires, such as the 


question on race/ethnicity added to the student questionnaire. 


e The PIRLS U.S. national public-use data files are available through 
NCES. The PIRLS U.S. national dataset can be downloaded from 
nces.ed.gov/surveys/pirls/datafiles.asp. Unlike the international 
files, these U.S. data files include the U.S.-specific adaptations made to 
questionnaire items, additional questions added to the school and student 


questionnaires, and some restricted variables designated by the IEA. 


e The PIRLS U.S. national restricted-use data files are available through 
NCES. Access to these files may be obtained by completing a restricted-use 
license agreement with NCES. The restricted-use data files are provided 
only on CD. These data files contain supplemental link files that link PIRLS 
school ID numbers to the school ID numbers as they appear in the publicly 
available Common Core of Data (CCD) or the Private School Universe 
Survey (PSS). In addition, race/ethnicity is provided with all available 
categories and free or reduced-price lunch status is provided as a continuous 
variable. Because these data can reveal the identities of participating schools, 
the restricted-use data files are only made available to those who obtain a 


NCES testticted-use data license. Directions on how to obtain the license 


can be found at https://nces.ed.gov/pubsearch/licenses.asp. 


The most comprehensive treatment of the PIRLS international data, and hence of 
the U.S. international data file, is provided in the various PIRLS 2016 publications 
produced by the IEA, particularly the PIRLS 2016 User Guide for the International 
Database (Foy 2018). This publication should be used as the primary reference. The 
U.S. PIRLS and ePIRLS 2016 Technical Report and User’s Guide draws heavily on the 
international user’s guide for much of its data file-related content. This content is 
supplemented with detail on those aspects of the PIRLS 2016 data that were unique 
to the United States. 
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Sample Design 


The PIRLS 2016 sample design resembled the sample design of PIRLS 2006 in that 
(1) the target student population was defined as the set of all fourth-graders in the 
United States in both public and private schools, and (2) one or two classrooms were 
selected for each assessment in each sampled school. In 2011, PIRLS was 
administered at the same time as the TIMSS. This resulted in a higher number of 
sampled schools than would have otherwise been experienced in a year when PIRLS 
was conducted on its own. The PIRLS 2016 school sample was drawn for the United 
States in November 2014. The sample design followed international requirements as 
described in Martin, Mullis, and Hooper (2017). 


The U.S. sample again used a two-stage design—a stratified systematic sample of 
schools with sampling probabilities proportional to size (PPS) and then classes 
within sampled schools. All students in sampled classrooms were selected for 
assessment. In 2016, all schools selected for PIRLS were also selected to participate 
in ePIRLS. The school sampling frame is described in section 2.1. The selection of 
schools in the first sampling stage is described in section 2.2 and the selection of 


classes and students in section 2.3. 


School Sampling Frame 


The school sampling frame was constructed using the 2011-12 CCD 
(https://nces.ed.gov/ccd/) and 2009-10 PSS (https://nces.ed.gov/surveys/pss/). 
These two national databases were developed by NCES and provide information 
about all schools in the United States. 


Eligible schools in the sampling frame included schools operating in the 50 states 
and the District of Columbia, Department of Defense (DoD) domestic schools, and 
Bureau of Indian Education schools that offer fourth-grade instruction to one or 
mote students. Schools in Puerto Rico and U.S. territories, DoD schools overseas, 
adult education institutions with no fourth-grade students, schools offering 
temporary instruction such as hospitals or treatment facilities, and non-education 
institutions (e.g., home schools, correspondence schools) were ineligible for the 
study. 
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Table 2-1 presents frame tabulations of the number of schools containing fourth- 


grade students. 


Table 2-1. Number and percentage of students and schools included in the U.S. PIRLS school 
sampling frame, by grade span: 2016 
Grade span Students Percent Schools Percent 
Total 3,989,251 100.0 69,235 100.0 
Grades PK/KG/1-5 1,979,349 49.6 25,917 37.4 
Grades PK/KG/1-6 676,132 16.9 12,073 17.4 
Grades PK/KG/1-8 472,844 11.9 15,248 22.0 
Grades PK/KG/1-12 135,372 3.4 6,344 9.2 
Other 725,554 18.2 9,653 13.9 


NOTE: KG = kindergarten; PK = prekindergarten. The “Other” grade span includes all additional grade spans. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


2.2 


2.2.1 


School Sample Selection 


The target population for schools, the units selected in the first stage of sampling, is 
defined as schools in the 50 United States and the District of Columbia providing 


instruction to one or more students in the fourth grade. 


Schools were sampled using a stratified systemic sample of schools with sampling 
probabilities proportional to size. The measure of size used was school fourth-grade 
enrollment, as measured on the sampling frame. Since the sampling frame contains 
data from 2009-2010 for private schools and 2011-2012 for public schools this 
measure of size is an estimate of the size measure for 2015-2016. This method 
reduced the chance of selection for smaller schools, which improved cost efficiency 
by increasing the average number of students per sampled school. Further, if 
students in schools with enrollments of only a few students were selected, they 
would have very large sampling weights, which could yield unstable variance 


estimates. 


The sample of 176 schools was allocated to strata in approximate proportion to the 
relative number of students in each stratum. High-poverty schools were oversampled 
by selecting 50 high-poverty schools, a slight deviation from proportional allocation 
to the sampling strata. 


Frame Stratification 


The school sampling frame was explicitly stratified by three categorical stratification 


variables: 
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e@ school control (public or private); 
e@ Census region (Northeast, Midwest, South, West);? and 
@ poverty level (high or low).° 


This resulted in 12 explicit strata. Within each explicit stratum, the frames were 
implicitly stratified (that is, sorted via hierarchical serpentine sorting) by two 


categorical stratification variables: 
e locale (city, suburb, town, or rural); and 
@ minority status (above or below 15 percent of the student population). 


The order of the stratification is not given due to confidentiality concerns. A third 
variable, Grade 4 enrollment, was also used in the serpentine sort to assist in the 


selection of substitute schools, described below. 


Substitute Schools 


Although efforts were made to secure the participation of all schools selected, it was 
anticipated at the time of sampling that not all schools would choose to participate. 
Therefore, as each school was selected for a sample, the two neighboring schools in 


the sampling frame were designated as substitute schools. 


The first school following the sampled school was the first substitute, and the first 
school preceding it was the second substitute. If an original school refused to 
participate, the first substitute was then contacted. If that school also refused to 


participate, the second substitute was then contacted. 


There were several constraints on the assignment of substitutes. One sampled school 
was not allowed to be a substitute for another, and a given school could not be 


assigned to be a substitute for more than one sampled school. Furthermore, 


> The Northeast region consists of Connecticut, Maine, Massachusetts, New Hampshire, New Jersey, 
New York, Pennsylvania, Rhode Island, and Vermont. The Midwest region consists of Illinois, 
Indiana, lowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South 
Dakota, and Wisconsin. The South region consists of Alabama, Arkansas, Delaware, the District of 
Columbia, Florida, Georgia, Kentucky, Louisiana, Maryland, Mississippi, North Carolina, Oklahoma, 
South Carolina, Tennessee, Texas, Virginia, and West Virginia. The West region consists of Alaska, 
Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oregon, Utah, 
Washington, and Wyoming. 

6 High-poverty schools are defined as having 76 percent or more of the students eligible for free or 
reduced-price lunch (FRPL) and low-poverty schools have less than 76 percent of students eligible for 
FRPL. Private schools are all classified as low poverty because no FRPL information is available. 
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2.3 


2.4 


substitutes were required to be in the same explicit stratum as the sampled school. If 
the sampled school was the first or last school in the stratum, then the second school 
following or preceding the sampled school was identified as the substitute. If the first 
substitute school did not have the same implicit stratification values as the sampled 
school, the first and second substitute were switched. Under these rules, it was 
possible to identify two substitutes for each sampled school. 


Selection of Classrooms and Students 


The second sampling stage consisted of selecting intact fourth-grade classes within 
each participating school. Schools were asked to list all classes containing fourth- 
grade students, indicating the number of fourth-grade students in the class and 
whether it was a “special class” in which all or most of the students were learning 
disabled or classified as having limited English proficiency. Because PIRLS does not 
provide accommodations, classrooms were excluded from the subsequent classroom 


sampling if all or most of the students were learning disabled. 


Classrooms with fewer than 15 students were collapsed into pseudo-classrooms so 
that each classroom in the school’s classroom sampling frame had at least 20 students. 
Because classrooms were sampled with equal probability within schools, small 
classrooms had the same probability of selection as large classrooms; hence, pseudo- 
classrooms were created for the purposes of classroom sampling, in which small 
classrooms were joined to reach a larger student count. These pseudo-classtrooms were 
treated as single classes in the class sampling process. Following class sampling, the 
pseudo-classroom combinations were dissolved, and the small classes involved 
retained their own identity. In this way, data on students, teachers, and classroom 


practices were linked in small classes in the same way as with larger classes. 


An equal probability sample of one to two classrooms (including pseudoclassrooms) 
was identified from the classroom frame for the school. In schools where there was 

only one classroom, this classroom was selected with certainty. For PIRLS 2016, 12 

pseudo-classrooms were created prior to classroom sampling, with 7 of these being 

selected in the final classroom sample. All students in sampled classrooms and 


pseudo-classrooms were selected for assessment. 


Tabulations Within Subgroups for Frame and Sample 


This section provides an overview of the fourth-grade frame and sample distribution 


by each of the stratification variables, showing that the PPS sampling and 
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stratification worked effectively: the sample percentage of schools was close to the 
measure-of-size percentage of the frame for all the implicit strata. The results are 
shown for Census region (table 2-2); poverty level (table 2-3); school control (table 2-4); 
locale (table 2-5); minority status (table 2-6); and Census region, poverty level, and 
school control (table 2-7). Each table provides the distribution of fourth-grade 


students (total measure of size) and schools in the sampling frame, as well as the 


distribution of schools in the sample. 


Table 2-2. Number and percentage of students and schools included in the U.S. PIRLS school 
sampling frame and number and percentage of schools in the sample, by Census 


region: 2016 
Frame Sample 
Number Number Number 
Census region of students Percent of schools Percent of schools Percent 
Total 3,989,251 100.0 69,235 100.0 176 100.0 
Northeast 646,256 16.2 11,867 17.1 29 16.5 
Midwest 856,259 21.5 17,457 25.2 37 21.0 
South 1,529,010 38.3 23,291 33.6 69 39.2 
West 957,726 24.0 16,620 24.0 41 23:3 


NOTE: For definitions of Census regions, see https://www2.census.gov/geo/docs/maps-data/maps/req_div.txt. 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


Table 2-3. Number and percentage of students and schools included in the U.S. PIRLS school 
sampling frame and number and percentage of schools in the sample, by poverty 


level: 2016 
Frame Sample 
Number Number Number 
Poverty level of students Percent of schools Percent of schools Percent 
Total 3,989,251 100.0 69,235 100.0 176 100.0 
High 832,265 20.9 12,411 17.9 50 28.4 
Low 3,156,986 79.1 56,824 82.1 126 71.6 


NOTE: For public schools, high poverty is defined as having 76 percent or more of the students eligible for free or reduced-price lunch, and 
low poverty is defined as having less than 76 percent eligible. Because no data were available for private schools, all private schools are 


categorized as low poverty. 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 
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Table 2-4. Number and percentage of students and schools included in the U.S. PIRLS school 
sampling frame and number and percentage of schools in the sample, by school 


control: 2016 


Frame Sample 
Number Number Number 
School control of students Percent of schools Percent of schools Percent 
Total 3,989,251 100.0 69,235 100.0 176 100.0 
Private 323,482 8.1 18,105 26.2 12 6.8 
Public 3,665,769 91.9 51,130 73.8 164 93.2 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


Table 2-5. Number and percentage of students and schools included in the U.S. PIRLS school 
sampling frame and number and percentage of schools in the sample, by locale: 2016 


Frame Sample 
Number Number Number 
Locale of students Percent of schools Percent of schools Percent 
Total 3,989,251 100.0 69,235 100.0 176 100.0 
City 1,205,700 30.2 21,021 30.4 59 33.5 
Suburb 1,391,223 34.9 20,875 30.2 58 33.0 
Town 441,385 11.1 7,453 10.8 21 11.9 
Rural 950,943 23.8 19,886 28.7 38 21.6 


NOTE: For definitions of these urbancentric locales, see https://nces.ed.gov/surveys/urbaned/definitions.asp. 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


Table 2-6. Number and percentage of students and schools included in the U.S. PIRLS school 
sampling frame and number and percentage of schools in the sample, by minority 


status: 2016 


Frame Sample 
Number Number Number 
Minority status of students Percent of schools Percent of schools Percent 
Total 3,989,251 100.0 69,235 100.0 176 100.0 
Below 15 percent 911,824 22.9 21,535 31.1 35 19.9 
15 percent or above 3,077,427 77.1 47,700 68.9 141 80.1 


NOTE: Minority status refers to the percentage of Black, Hispanic, Asian and Pacific Islander, and American Indian and Alaska Native 
students. Black includes African American and Hispanic includes Latino. Racial categories exclude Hispanic origin. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 
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Table 2-7. Number and percentage of students and schools included in the U.S. PIRLS school 
sampling frame, by Census region, poverty level, and school control: 2016 


Frame Sample 

Census Poverty School Number Number Number 
region level control of students Percent of schools Percent of schools Percent 

Total 3,989,251 100.0 69,235 100.0 176 100.0 
Northeast High Public 122,497 3.1 1,623 2.3 8 4.5 
Midwest High Public 146,276 3.7 2,680 3.9 9 5.1 
South High Public 406,354 10.2 5,635 8.1 24 13.6 
West High Public 157,138 3.9 2,473 3.6 9 5.1 
Northeast Low Private 70,356 1.8 3,914 5.7 3 1.7 
Midwest Low Private 86,064 2.2 4,880 7.0 3 Tf 
South Low Private 106,130 2.7 5,760 8.3 4 2.3 
West Low Private 60,932 1.5 3,551 9A 2 1.4 
Northeast Low Public 453,403 11.4 6,330 9.1 18 10.2 
Midwest Low Public 623,919 15.6 9,897 14.3 25 14.2 
South Low Public 1,016,526 25.5 11,896 17.2 41 23.3 


West Low Public 739,656 18.5 10,596 15.3 30 17.0 


NOTE: For definitions of Census regions, see https://www2.census.gov/geo/docs/maps-data/maps/req_div.txt. For public schools, high 
poverty is defined as having 76 percent or more of the students eligible for free or reduced-price lunch, and low poverty is defined as having 
less than 76 percent eligible. Because no data were available for private schools, all private schools are categorized as low poverty. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Participation Rates and Nonresponse 
Bias 


To minimize the potential for response biases, the IEA developed participation or 
response rate standards that apply to all participating education systems and govern 
whether data are included in the PIRLS 2016 international datasets and the way in 
which aggregate statistics are presented in the international reports. These standards 
were set using composites of participation rates at the school, classroom, and student 
levels, and were calculated with and without the inclusion of substitute schools that 


were selected to replace schools refusing to participate. 


The standards take the following two forms, distinguished primarily by whether 
meeting the school participation rate of 85 percent requires the counting of 


substitute schools: 


Category 1: Met requirements. Education systems that met all of the following 
conditions were considered to have fulfilled the IEA requirements: (1) a 
minimum school participation rate of 85 percent, based on original sampled 
schools only; and (2) a minimum classroom participation rate of 95 percent, from 
both original and substitute schools; and (3) a minimum student participation 


rate of 85 percent, from both original and substitute schools. 


Category 2: Met requirements after substitutes. In the case of education 
systems that did not meet the category 1 requirements, and as long as at least 50 
percent of schools in the original sample participated, an education system’s data 
were considered acceptable if the following requirements were met: a minimum 
combined school, classroom, and student participation rate of 75 percent, based 
on the product of the participation rates described above. That is, the product of 
(1), (2), and (3), as defined in the category 1 standard, must be greater than or 
equal to 75 percent. 


Education systems satisfying the category 1 standard were included in the 
international tabular presentations without annotation. Those able to satisfy only 
the category 2 standard were included as well but were annotated to indicate their 


response rate status. 
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3.1 Exclusions 


The national defined target population is described in Methods and Procedures in PIRLS 
2016 (Martin, Mullis, and Hooper 2017). All schools and students excluded from this 
population are referred to as the “excluded population.” Exclusions could occur at 
the school level, with entire schools being excluded, or within schools, with specific 
students or entire classrooms excluded. PIRLS 2016 did not provide 
accommodations for students with disabilities or students who were unable to read 


or speak the language of the test. 


3.1.1 School Exclusions 


Countries could exclude schools that 
@ were geographically inaccessible; 
e@ were of extremely small size; 


@ offered a curriculum or school structure radically different from the 


mainstream educational system; or 


e@ provided instruction only to students in the excluded categories defined 


under “‘within-school exclusions,” such as schools for the blind. 


3.1.2 Within-School Exclusions 


Countries were asked to adapt the following international within-school exclusion 


tules to define excluded students: 


Students with intellectual disabilities. Students who, in the professional 
opinion of the school principal or other qualified staff members, were considered 
to be intellectually disabled or who had been tested psychologically as such. This 
included students who were emotionally or mentally unable to follow even the 
general instructions of the test. Students were not to be excluded solely because 


of poor academic performance or normal disciplinary problems. 


Students with functional disabilities. Students who were permanently 
physically disabled in such a way that they could not perform in the PIRLS 
testing situation. Functionally disabled students who were able to respond were 


included in the testing. 
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Non-native-language speakers. Students who were unable to read or speak 


the language(s) of the test and were unable to overcome the language barrier of 


the test. Typically, a student who had received less than 1 year of instruction in 


the language(s) of the test was excluded. 


3.2 PIRLS Participation Rates of U.S. Schools, Classrooms, 


and Students 


The raw numbers on which the various participation rates were based, along with the 


participation rates themselves, are shown in tables 3-1 and 3-2. To explain how to 


interpret these participation rates, subsections 3.2.1 through 3.2.5 describe a 


complete interpretation of the numbers. 


Table 3-1. Number of U.S. schools, classrooms, and students participating in PIRLS, and 
participation rates, for fourth grade: 2016 


Participation rates 


Participation status Number Unweighted Weighted 
Schools 

Sampled 176 

Excluded and ineligible 4 

Eligible 172 

Participating 131 

Substitutes 27 
Participating (all schools) 158 92 92 
Classrooms in participating schools 

Total 708 

Excluded 18 

Eligible 690 

Sampled 215 
Participating 215 100 100 
Students in participating schools 

Sampled 5,056 

Excluded 175 

Eligible 4,722 

Absent 297 
Assessed 4,425 94 94 


NOTE: National Center for Education Statistics standards (Standard 1-3-8) indicate that participation rates should be calculated without 
including substitute schools since substitute schools do not have an independent probability of selection (Seastrom 2014). However, the 
participation rates shown in this table are those reported by PIRLS and do include substitute schools in the calculations. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS), 2016. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


| 20 | 3. PARTICIPATION RATES AND NONRESPONSE BIAS 


Table 3-2. Number of U.S. schools, classrooms, and students participating in ePIRLS, and 


participation rates, for fourth grade: 2016 


Participation rates 


Participation status Number Unweighted Weighted 
Schools 

Sampled 176 

Excluded and ineligible 4 

Eligible 172 

Participating 128 

Substitutes 25 
Participating (all schools) 153 89 89 
Classrooms in participating schools 

Total 701 

Excluded 18 

Eligible 683 

Sampled 208 
Participating 208 100 100 
Students in participating schools 

Sampled 4,884 

Excluded 175 

Eligible 4,554 

Absent 464 
Assessed 4,090 90 90 


NOTE: NCES standards (Standard 1-3-8) indicate that participation rates should be calculated without including substitute schools since 
substitute schools do not have an independent probability of selection (Seastrom 2014). However, the participation rates shown in this table 


are those reported by PIRLS and do include substitute schools in the calculations. 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS), 2016. 


3.2.1 Interpreting School Participation Rates 


Of the 176 schools sampled for PIRLS 2016 in the United States, four schools were 
matked ineligible because they had closed or did not have any enrolled fourth-grade 
students. Replacement schools are not to be used as a substitute for schools that are 


closed or ineligible due to lack of students in the target grade. 


Of the remaining 172 schools, a total of 131 originally sampled schools participated 
in PIRLS 2016, for an unweighted participation rate of 76 percent. The weighted 
school participation rate, which accounts for nonparticipation and unequal 


probability of selection due to stratification, was 75 percent. 


In addition to the 131 participating original schools, 27 replacement schools also 
participated, for a total of 158 participating schools. The weighted and unweighted 


school participation rates with replacement schools were 92 percent. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


3. PARTICIPATION RATES AND NONRESPONSE BIAS | 21 | 


All schools selected for PIRLS were also asked to participate in ePIRLS. Five of the 
158 participating PIRLS schools declined to also participate in ePIRLS. The 
weighted and unweighted school participation rates for ePIRLS in the final sample 


with replacement schools were 89 percent. 


3.2.2 Interpreting Classroom Participation Rates 


In accord with the international requirements, schools agreeing to participate were 
asked to list their fourth-grade reading classes as the basis for sampling at the 
classroom level. Schools appeared to be able to identify classes in this way without 
any problems. A total of 708 classrooms were identified from the 158 participating 
schools. Schools were then given the opportunity to identify special classes—classes 
containing all, or a majority of, students with intellectual or functional disabilities, or 
students who were non-native-language speakers. Although these classes were 
regarded as eligible, the students as a group were treated as excluded because, in the 
opinion of the school, their disabilities or language capabilities would render 
meaningless theit performance on the assessment. A total of 18 classrooms were 
excluded in this way. This left a pool of 690 eligible classrooms from which a sample 


of 215 classrooms was drawn. All selected classrooms participated in PIRLS. 


Subsequently, schools were asked to list the students in each of the 215 sampled 
classrooms at the fourth grade, along with the teachers who taught reading to these 
students. Schools were then given the opportunity to identify particular students not 
suited to take the test because of functional or intellectual disabilities or because they 


were non-native-language speakers. 


3.2.3 Interpreting Student Participation Rates 


A total of 5,056 students were selected to participate in PIRLS from the 158 
participating schools. Of those students, a total of 175 students (less than one 
percent) were excluded based on the disability categories described earlier (17 were 
excluded due to a functional disability, 122 due to an intellectual disability, and 36 


due to language issues). 


Of the remaining 4,722 students in the eligible sample, some 4,425 students 
participated, corresponding to weighted and unweighted response rates of 94 
percent. From the 153 schools that participated in ePIRLS, 4,090 students 
participated, for weighted and unweighted response rates of 90 percent. 
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Combined Participation Rates 


The PIRLS 2016 sample met the combined school, classroom, and student weighted 
participation rate standard of 75 percent used by PIRLS in situations in which it was 
necessary to recruit substitute schools. The application of international guidelines 
means, however, that U.S. statistics describing fourth-grade students in PIRLS are 
annotated in international reports to indicate that coverage of the defined student 
population was less than the IEA standard of 95 percent and that participation rates 


were met only after substitute schools were included. 


Teacher and Administrator Response Rates 


The school administrator at each participating school and the reading teacher for 
each selected classroom were asked to complete a questionnaire. Of the 158 
administrators asked to complete the questionnaire, 150 responded, for a weighted 
and unweighted response rate of 95 percent. Of the 215 teachers asked to 
participate, 207 responded, for a weighted and unweighted response rate of 96 


percent. 


Exclusions in the U.S. National Samples 


As noted earlier, schools were given the opportunity to exclude any special classes 
among the total number of classes in the fourth grade. These classes were made up 
largely of students with functional or intellectual disabilities or students who were 
non-native-language speakers, as defined in section 3.1.2. Classes identified in this 
way were excluded from the class sampling procedure. Subsequently, schools were 
given the opportunity to exclude students from the sampled classes—essentially, 
students with functional or intellectual disabilities, or non-native-language-speaking 
students in the United States who had been mainstreamed. 


These procedures resulted in a (weighted) student exclusion rate of 4.85 percent for 
PIRLS and 4.94 percent for ePIRLS. The desired exclusion rate is five percent or 


less. 
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Participation Rates for All Countries 


For comparable school, classroom, and student participation rates in other nations in 
PIRLS, see exhibits C-2 through C-5 in appendix C of PIRLS 2016 International 
Results in Reading (Mullis et al. 2017). 


Nonresponse Bias Analysis 


The accuracy of survey statistics is affected by both random and nonrandom errors. 
Random errors reduce the precision of survey estimates, and nonrandom errors may 
result in bias (i.e., estimates that do not converge to the true population parameter as 


the sample size increases without limit) or loss of precision. 


The sources of error in a survey are often dichotomized as sampling and 
nonsampling errors. Sampling error refers to the error that occurs because the survey 
is based on a sample of population members rather than the entire population. All 
other types of errors are nonsampling errors, including survey nonresponse (because 
of inability to contact sampling members, their refusal to participate in the study, 
etc.) and measurement errors, such as the errors that occur because the intent of 
sutvey questions was not clear to the respondent, because the respondent had 
insufficient knowledge to answer correctly, or because the data were not captured 


correctly (e.g., because of recording, editing, or data entry errors). 


NCES Statistical Standard 4-4-1 states that “Any survey stage of data collection with 
a unit or item response rate less than 85 percent must be evaluated for the potential 
magnitude of nonresponse bias before the data or any analysis using the data may be 
released. Estimates of survey characteristics for nonrespondents and respondents are 
required to assess the potential nonresponse bias” (Seastrom 2014). This section 
describes the nonresponse bias analysis conducted for U.S. PIRLS 2016. 


A school respondent is defined as a school that allowed its students to participate. 

The weighted response rate for the originally sampled schools was 76 percent. After 
including replacement schools, the response rate was 92 percent. All classes selected 
within schools participated. The weighted response rate for students in participating 
schools was 94 percent. Hence, a unit nonresponse bias analysis was conducted only 


at the school level. 


Item response rates were computed using final analysis weights. Item response rates 
were above 85 percent for all school variables but one. Per NCES Standard 4-4-3, 


nonresponse bias analyses were conducted for this variable (Seastrom 2014). 
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Weighted item response rates were above 85 percent for all teacher and student 


variables. 


The school nonresponse bias analysis for PIRLS was conducted for both the original 
sample and the final sample with replacement schools. Both samples were weighted 
using base weights, equal to the reciprocal of selection probabilities. For the final 
sample, base weights for substitute schools were set to equal the base weight of the 
school they replaced. These analyses describe the bias that may be present due to 
school nonpatticipation and any mitigation of the bias due to using substitute schools. 
An additional analysis was conducted for the final sample to evaluate any remaining 


bias after the base weights were adjusted to account for nonresponding schools. 


All analyses were conducted using SUDAAN software (Version 11; RTI International 
2017) for analyzing data for complex surveys so that standard errors and related 
statistics correctly account for the sample design. The approach used for estimating 
standard errors was Taylor series linearization. Note that PIRLS public-use files do not 
contain the design variables necessary for Taylor series linearization variance estimation 
due to confidentiality concerns; instead, users are provided with tools for estimating 


standard errors using jackknife replication weights (described in section 5.11.2). 


3.4.1 Methodology 


The bias in an estimated mean based on respondents, Vp is the difference between 


the expected value of this mean and the target parameter, 2, the population mean. 
Analysts can estimate the target parameter for variables that are observed for both 


respondents and nonrespondents as follows: 7 =(1—7)V¥p +7Vnp, where 7 is the 


weighted unit (or item) nonresponse rate. For variables that are from the frame 
rather than from the sample, analysts can estimate z without sampling error. They 


can then estimate bias as the difference between the respondent mean and the full- 
sample mean: B (Vr) =Vp—Z. Equivalently, bias can be estimated as the difference 
between the mean for respondents and the mean for nonrespondents, multiplied by 


the weighted nonresponse rate: B (Vr)=1(Vr-Ynr)- 


Relative bias provides a measure of the magnitude of the bias relative to the sample 
mean and is estimated as RB(¥p) = BWR) / 7. Effect size, as defined by Cohen 


(1988), is another measure of potential nonresponse bias. For continuous variables, it 
is computed as the estimated bias divided by the full-sample standard deviation: 


B(DR)/ Gy. For categorical variables, it is computed as \/2; (Po; py / Poj> 


where Pg; is the full-sample proportion in category i, and pj; is the respondent 
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proportion in category 7. Effect sizes can be used in combination with bias and 
relative bias estimates and significance tests to evaluate the potential for nonresponse 
bias. Cohen classified an effect size as “small” when it is about 0.10, as “medium” 


when it is about 0.30, and as “large” when it is about 0.50. 


Nontresponse bias can only be measured for variables known for both respondents 
and nonrespondents; hence, analyses involve variables available on the sampling 
frame. The extent to which the analyses accurately detect bias depends on the degree 
to which the school characteristics available on the frame are related to other survey 
items. The variables available for all schools in either the 2014-15 CCD or 2013-14 
PSS that were used in the bias analysis include some of the same school 


characteristics used to stratify the sampling frame (described in section 2.2.1): 
e school control (public or private); 
e locale (city, suburb, town, or rural); 
e Census region (Northeast, Midwest, South, West); and 
@ poverty level (high or low). 
In addition, the following school characteristics were used: 
@ percent free or reduced-price lunch eligibility (public schools only); 
e fourth-grade enrollment; 
e@ total school enrollment; and 


@ percentage of students in seven race/ethnicity categories (White non- 
Hispanic, Black non-Hispanic, Hispanic, Asian, American Indian or Alaska 


Native, Native Hawaiian or Pacific Islander, or Two or mote races). 


For continuous frame variables, estimates of bias, relative bias, and effect size were 
calculated. Bias was tested for statistical significance using ¢+tests. For categorical 
frame variables, bias and relative bias were estimated for each variable category, 
effect size was calculated, and chi-square tests for independence’ between response 


status and frame characteristic were conducted. 


The bias analyses for the original and final samples, prior to weight adjustment for 


nonresponse, were supplemented by logistic regressions predicting response 


7 The Wald chi-square test, analogous to the Pearson chi-square test for nonsurvey data, was used. 
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propensity from the available frame variables. These were used to examine the joint 
relationship of the frame variables with response status. Because the race/ethnicity 
percentages added up to 100 percent for each school and were linearly dependent, 
they could not all be included in the same model, so two logistic models were run. 
The first included all race/ethnicity categories excluding percent White non- 
Hispanic; and the second also excluded percent White non-Hispanic but used the 
summed percentage of the other six race/ethnicity categories. For categorical 


predictor variables, the largest category was assigned to be the reference level. 


School Nonresponse Bias Analysis 


This section summarizes the results of the three school nonresponse bias analyses: 
(1) evaluation of bias due to school nonresponse in the original sample; 

(2) evaluation of bias due to school nonresponse after replacement schools were 
used; and (3) bias due to school nonresponse after accounting for replacement 


schools and nonresponse weight adjustments. 


Bias analysis for original sample 


This analysis evaluated the potential for bias due to school nonresponse. Out of the 
original sample of 172 eligible schools, 131 agreed to participate. 


Tables 3-3 through 3-6 provide estimates of bias, relative bias, and effect size, as well 
as the p-values from chi-square tests of independence. Overall the differences 
between the eligible sample and participating sample were small to moderate. One 
categorical variable, Census region, had a statistically significant relationship with 
participation status, shown in table 3-3. Compared to the eligible sample, the 
participating schools included fewer schools in the Midwest and West Census 
regions and more in the South region. Among continuous variables, the differences 
between the means for the eligible and participating samples were statistically 
significant for percentage of Black non-Hispanic students (table 3-5) and percentage 
of students eligible for free and reduced-price lunch (table 3-6). The means for both 
variables were slightly higher among participating schools. Bias for free and reduced- 
price lunch is computed using the 162 eligible public schools, out of which 123 
participated. 


When the relationships between the frame variables and participation status were 
considered jointly in a logistic model, many coefficient estimates were close to zero, 
as seen in table 3-7 and table 3-8. The only coefficient statistically different from zero 
was the coefficient for the South Census region in the second model (table 3-7). 
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Table 3-3. Percentage distribution of eligible and participating schools in the U.S. PIRLS original 
sample and nonresponse bias indicators, by selected school characteristics: 2016 


Sample schools 


Eligible Participating Relative Chi-square Effect 
School characteristic (percent) (percent) Bias bias p-value size 
School control 
Public 80.15 77.84 -2.31 -0.03 0.4063 0.06 
Private 19.85 22.16 2.31 0.12 
Locale 
City 35.40 39.85 4.45 0.13 0.0787 0.15 
Suburb 33.61 26.59 -7.02 -0.21 
Town 13.93 15.93 2.00 0.14 
Rural 17.06 17.63 0.57 0.03 
Census region 
Northeast 15.48 15.38 -0.11 -0.01 0.0002 0.21 
Midwest 26.34 20.45 -5.89 -0.22 
South 37.94 47.32 9.38 0.25 
West 20.24 16.85 -3.39 -0.17 
Poverty level 
High 72.84 69.32 -3.52 -0.05 0.0587 0.08 
Low 27.16 30.68 3.52 0.13 


NOTE: A high-poverty public school is defined as one in which 76 percent or more of the students are eligible for participation in the free or 
reduced-price lunch program, and low poverty is defined as having less than 76 percent eligible. All private schools are treated as low- 
poverty schools. For definitions of Locale, see https://nces.ed.gov/surveys/urbaned/definitions.asp. For definitions of Census regions, see 
https://www2.census.gov/qeo/docs/maps-data/maps/req _div.txt. Eligible schools are those with any fourth-grade students. Participating 
schools are those in which students were assessed. Bias is estimated as the difference between the participant and the eligible sample 
estimates. The relative bias is calculated as the estimated bias divided by the eligible sample estimate. Effect size is calculated as the square 
root of the sum over categories of the squared bias estimates over full-sample means. The chi-square p-value is the result of a test for 
independence between participation status and the variable indicated. All calculations use school sampling weights. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


Table 3-4. Mean enrollment of eligible and participating schools in the U.S. PIRLS original sample 
and nonresponse bias indicators: 2016 


Sample schools 


Eligible Participating Relative ttest Effect 
Enrollment level (mean) (mean) Bias bias p-value size 
Total school 431.06 426.17 4.88 0.01 0.6534 0.02 
Fourth grade 65.67 63.92 1.75 0.03 0.3637 0.04 


NOTE: Eligible schools are those with any fourth-grade students. Participating schools are those in which students were assessed. Bias is 
estimated as the difference between the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias 
divided by the eligible sample estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value 
reported is the result of the significance test for the difference between bias and zero. All calculations use school sampling weights. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA) Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Table 3-5. Mean percentage of students in eligible and participating schools in the U.S. PIRLS 
original sample and nonresponse bias indicators, by race/ethnicity: 2016 


Sample schools 


Eligible Participating Relative ttest Effect 
Race/ethnicity (percent) (percent) Bias bias p-value size 
White, non-Hispanic 50.87 49.37 1.50 0.03 0.4125 0.05 
Black, non-Hispanic 17.29 19.89 -2.60 -0.15 0.0276 0.10 
Hispanic 19.94 18.25 1.68 0.08 0.2134 0.07 
Asian 3.98 3.60 0.38 0.10 0.3268 0.04 
American Indian or Alaska Native 1.86 2.29 -0.43 -0.23 0.1044 0.05 
Native Hawaiian or Pacific Islander 0.23 0.19 0.04 0.19 0.3861 0.06 
Two or more races 3.92 3.95 -0.03 -0.01 0.8838 0.01 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude Hispanic origin. Eligible schools are those 
with any fourth-grade students. Participating schools are those in which students were assessed. Bias is estimated as the difference between 
the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias divided by the eligible sample 
estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value reported is the result of the 
significance test for the difference between bias and zero. All calculations use school sampling weights. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


Table 3-6. Mean percentage of students eligible for free or reduced-price lunch, in eligible and 
participating public schools in the U.S. PIRLS original sample and nonresponse bias 
indicators: 2016 


Sample schools 


Eligible Participating Relative ttest Effect 

School characteristic (percent) (percent) Bias bias p-value size 
Percentage of students eligible for free or 

reduced-price lunch 59.33 62.63 -3.31 -0.06 0.0291 0.09 


NOTE: Eligible schools are those with any fourth-grade students. Participating schools are those in which students were assessed. Bias is 
estimated as the difference between the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias 
divided by the eligible sample estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value 
reported is the result of the significance test for the difference between bias and zero. All calculations use school sampling weights. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Table 3-7. Logistic regression model parameter estimates (with six race/ethnicity variables) using 
the U.S. PIRLS original sample and nonresponse bias indicators: 2016 


Parameter Standard t test for Ho: 
Parameter estimate error parameter = 0 p-value 
Intercept 1.37 1.11 1.2379 0.2182 
Locale: City 0.22 0.87 0.2506 0.8025 
Locale: Suburb -1.08 0.82 -1.3141 0.1913 
Locale: Town 0.60 1.07 0.5602 0.5764 
School control: Private 0.73 0.93 0.7837 0.4347 
Poverty level: High 1.19 0.71 1.6734 0.0969 
Census region: Northeast -0.22 0.89 -0.2452 0.8067 
Census region: Midwest -1.56 0.97 -1.6098 0.1101 
Census region: South 1.44 0.85 1.6811 0.0953 
Total school enrollment 0.00 0.00 0.0664 0.9472 
Fourth-grade enrollment 0.00 0.01 0.1309 0.8960 
Percent Black, non-Hispanic 0.01 0.01 0.6120 0.5417 
Percent Hispanic -0.02 0.01 -1.8251 0.0705 
Percent Asian 0.01 0.03 0.2245 0.8227 
Percent American Indian or Alaska Native 0.07 0.05 1.4477 0.1503 
Percent Native Hawaiian or Pacific Islander -0.20 0.17 -1.1914 0.2359 
Percent Two or more races 0.05 0.06 0.7457 0.4573 


NOTE: A high-poverty public school is defined as one in which 76 percent or more of the students are eligible for participation in the free or 
reduced-price lunch program, and low poverty is defined as having less than 76 percent eligible. All private schools are treated as low- 
poverty schools. For definitions of Locale, see https://nces.ed.gov/surveys/urbaned/definitions.asp. For definitions of Census regions, see 
https://www2.census.gov/geo/docs/maps-data/maps/reg _div.txt. The response variable is participation status. The largest categories for 
each variable were selected as the reference level. School sampling weights were used. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Table 3-8. Logistic regression model parameter estimates (with summed race/ethnicity 
percentages) using the U.S. PIRLS original sample and nonresponse bias indicators: 


2016 
Parameter Standard t test for Ho: 
Parameter estimate error parameter = 0 p-value 
Intercept 0.72 1.01 0.7167 0.4749 
Locale: City 0.55 0.79 0.6966 0.4874 
Locale: Suburb -0.73 0.73 -1.0096 0.3147 
Locale: Town 0.92 1.02 0.9011 0.3693 
School control: Private 0.57 0.92 0.6258 0.5326 
Poverty level: High 0.83 0.65 1.2728 0.2056 
Census region: Northeast 0.49 0.70 0.7024 0.4838 
Census region: Midwest -0.57 0.65 -0.8776 0.3819 
Census region: South 2.17 0.68 3.1877 0.0018 
Total school enrollment 0.00 0.00 0.3086 0.7582 
Fourth-grade enrollment -0.00 0.01 -0.3954 0.6933 
Summed race/ethnicity percentage -0.01 0.01 -0.5885 0.5573 


NOTE: A high-poverty public school is defined as one in which 76 percent or more of the students are eligible for participation in the free or 
reduced-price lunch program, and low poverty is defined as having less than 76 percent eligible. All private schools are treated as low- 
poverty schools. For definitions of Locale, see https://nces.ed.gov/surveys/urbaned/definitions.asp. For definitions of Census regions, see 
https://www2.census.gov/geo/docs/maps-data/maps/reg_div.txt. The response variable is participation status. The largest categories for 
each variable were selected as the reference level. School sampling weights were used. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


3.4.2.2 Bias analysis for final sample (with replacement schools) 


This analysis evaluated the potential for bias due to school nonresponse, after 
substituting nonresponding schools with replacement schools. Some replacement 
schools were also nonrespondents. If the first replacement school was a 
nonrespondent, then the second replacement school was contacted. In total, 28 first 
or second replacement schools participated, yielding 158 total participating schools 
out of 172 eligible schools. 


Compared to the original sample analysis described in section 3.4.2.1, estimates of 
bias, relative bias, and effect size were smaller for virtually all variables in the sample 
with replacement schools (table 3-9 through 3-12). Census region was the only 
variable remaining with a statistically significant relationship with participation status. 
Similar to the original sample, schools in the South Census region are represented at 
a higher rate among participating schools than among the overall eligible schools, 


while schools in other regions are represented at slightly lower rates. 


When the relationships between the frame variables and participation status were 
considered jointly in a logistic model, the results differ from the first analysis without 
replacement schools (tables 3-7 and 3-8). Census region no longer had any 


statistically significant coefficients in either model; however, in the first model 
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from zero, suggesting that schools with higher fourth-grade enrollment are 


associated with higher propensity to respond. In the second model (table 3-14), the 


coefficient for total school enrollment was statistically different from zero; however, 


the estimated coefficient is nearly indistinguishable from zero. 


Table 3-9. Percentage distribution of eligible and participating schools in the U.S. PIRLS final 


Sample schools 


sample, by selected school characteristics and nonresponse bias indicators: 2016 


Eligible Participating Relative Chi-square Effect 
School characteristic (percent) (percent) Bias bias p-value size 
School control 
Public 80.15 79.35 -0.80 -0.01 0.3771 0.02 
Private 19.85 20.65 0.80 0.04 
Locale 
City 35.13 35.69 0.56 0.02 0.6219 0.03 
Suburb 32.99 31.74 -1.25 -0.04 
Town 13.72 14.50 0.77 0.06 
Rural 18.15 18.07 -0.08 -0.00 
Census region 
Northeast 15.48 15.26 -0.22 -0.01 0.0085 0.07 
Midwest 26.34 25.23 -1.11 -0.04 
South 37.94 41.29 3.35 0.09 
West 20.24 18.22 -2.02 -0.10 
Poverty level 
High 72.78 72.65 -0.12 -0.00 0.9019 0.00 
Low 27.22 27.35 0.12 0.00 


NOTE: A high-poverty public school is defined as one in which 76 percent or more of the students are eligible for participation in the free or 
reduced-price lunch program, and low poverty is defined as having less than 76 percent eligible. All private schools are treated as low- 
poverty schools. For definitions of Locale, see https://nces.ed.gov/surveys/urbaned/definitions.asp. For definitions of Census regions, see 
https://www2.census.gov/geo/docs/maps-data/maps/req_div.txt. Eligible schools are those with any fourth-grade students. Participating 


schools are those in which students were assessed. Bias is estimated as the difference between the participant and the eligible sample 
estimates. The relative bias is calculated as the estimated bias divided by the eligible sample estimate. Effect size is calculated as the square 
root of the sum over categories of the squared bias estimates over full-sample means. The chi-square p-value is the result of a test for 
independence between participation status and the variable indicated. All calculations use school sampling weights. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER’S GUIDE 


| 32 | 3. PARTICIPATION RATES AND NONRESPONSE BIAS 


Table 3-10. Mean enrollment of eligible and participating schools in the U.S. PIRLS final sample 
and nonresponse bias indicators: 2016 


Sample schools 


Eligible Participating Relative ttest Effect 
Enrollment level (mean) (mean) Bias bias p-value size 
Total school 431.63 429.14 2.48 0.01 0.6153 0.01 
Fourth grade 64.16 64.33 -0.18 -0.00 0.8319 0.00 


NOTE: Eligible schools are those with any fourth-grade students. Participating schools are those in which students were assessed. Bias is 
estimated as the difference between the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias 
divided by the eligible sample estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value 
reported is the result of the significance test for the difference between bias and zero. All calculations use school sampling weights. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


Table 3-11. Mean percentage of students in eligible and participating schools in the U.S. PIRLS 
final sample and nonresponse bias indicators, by race/ethnicity: 2016 


Sample schools 


Eligible Participating Relative ttest Effect 
Race/ethnicity (percent) (percent) Bias bias p-value size 
White, non-Hispanic 51.58 51.84 -0.26 -0.01 0.7733 0.01 
Black, non-Hispanic 17.05 17.81 -0.76 -0.04 0.1327 0.03 
Hispanic 19.37 18.58 0.79 0.04 0.3026 0.03 
Asian 4.10 3.64 0.46 0.11 0.1268 0.05 
American Indian or Alaska Native 1.87 2.01 -0.14 -0.07 0.0922 0.02 
Native Hawaiian or Pacific Islander 0.18 0.17 0.01 0.04 0.3779 0.02 
Two or more races 3.88 3.84 0.04 0.01 0.6557 0.01 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude Hispanic origin. Eligible schools are those 
with any fourth-grade students. Participating schools are those in which students were assessed. Bias is estimated as the difference between 
the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias divided by the eligible sample 
estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value reported is the result of the 
significance test for the difference between bias and zero. All calculations use school sampling weights. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


Table 3-12. Mean percentage of students eligible for free or reduced-price lunch, in eligible and 
participating public schools in the U.S. PIRLS final sample and nonresponse bias 
indicators: 2016 


Sample schools 


Eligible Participating Relative ttest Effect 

School characteristic (percent) (percent) Bias bias p-value size 
Percentage of students eligible for free or 

reduced-price lunch 58.66 59.45 -0.79 -0.01 0.3146 0.02 


NOTE: Eligible schools are those with any fourth-grade students. Participating schools are those in which students were assessed. Bias is 
estimated as the difference between the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias 
divided by the eligible sample estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value 
reported is the result of the significance test for the difference between bias and zero. All calculations use school sampling weights. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Table 3-13. Logistic regression model parameter estimates (with six race/ethnicity variables) using 
the U.S. PIRLS final sample and nonresponse bias indicators: 2016 


Parameter Standard t test for Ho: 

Parameter estimate error parameter = 0 p-value 
Intercept 2.80 1.55 1.8058 0.0734 
Locale: City 0.44 1.17 0.3723 0.7103 
Locale: Suburb -0.06 0.93 -0.0666 0.9470 
Locale: Town 0.83 1.35 0.6128 0.5411 
School control: Private 1.29 1.68 0.7691 0.4433 
Poverty level: High 0.04 1.09 0.0356 0.9717 
Census region: Northeast’ -0.29 0.93 -0.3101 0.7570 
Census region: Midwest’ -1.14 0.97 -1.1753 0.2422 
Total school enrollment -0.00 0.00 -1.8551 0.0660 
Fourth-grade enrollment 0.03 0.01 2.3977 0.0180 
Percent Black, non-Hispanic 0.02 0.03 0.8511 0.3964 
Percent Hispanic -0.02 0.02 -0.9413 0.3484 
Percent Asian -0.04 0.02 -1.8503 0.0667 
Percent American Indian or Alaska Native 0.50 0.62 0.8000 0.4253 
Percent Native Hawaiian or Pacific Islander -0.24 0.36 -0.6809 0.4972 
Percent Two or more races -0.10 0.07 -1.4941 0.1378 
‘ There are no nonparticipating schools in the South Census region, so the participating South schools were combined with schools in the 


West region. 

NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude Hispanic origin. A high-poverty public 
school is defined as one in which 76 percent or more of the students are eligible for participation in the free or reduced-price lunch program, 
and low poverty is defined as having less than 76 percent eligible. All private schools are treated as low-poverty schools. For definitions of 
Locale, see https://nces.ed.gov/surveys/urbaned/definitions.asp. For definitions of Census regions, see 
https://www2.census.gov/qeo/docs/maps-data/maps/req _div.txt. The response variable is participation status. The largest categories for 
each variable were selected as the reference level. School sampling weights were used. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Table 3-14. Logistic regression model parameter estimates (with summed race/ethnicity 


percentages) using the U.S. PIRLS final sample and nonresponse bias indicators: 2016 


Parameter Standard t test for Ho: 

Parameter estimate error parameter = 0 p-value 
Intercept 3.06 1.47 2.0812 0.0395 
Locale: City 0.40 1.17 0.3392 0.7350 
Locale: Suburb -0.35 0.92 -0.3856 0.7005 
Locale: Town 0.87 1.45 0.6044 0.5467 
School control: Private 1.04 1.41 0.7325 0.4653 
Poverty level: High 0.68 0.93 0.7270 0.4686 
Census region: Northeast’ -0.63 0.86 -0.7408 0.4603 
Census region: Midwest! -1.17 0.75 -1.5558 0.1224 
Total school enrollment -0.00 0.00 -2.2069 0.0292 
Fourth-grade enrollment 0.02 0.01 1.8810 0.0624 
Summed race/ethnicity percentage -0.01 0.01 -0.8650 0.3887 
‘ There are no nonparticipating schools in the South Census region, so the participating South schools were combined with schools in the 
West region. 


NOTE: A high-poverty public school is defined as one in which 76 percent or more of the students are eligible for participation in the free or 
reduced-price lunch program, and low poverty is defined as having less than 76 percent eligible. All private schools are treated as low- 
poverty schools. For definitions of Locale, see https://nces.ed.gov/surveys/urbaned/definitions.asp. For definitions of Census regions, see 
https://www2.census.gov/geo/docs/maps-data/maps/reg _div.txt. The response variable is participation status. The largest categories for 


each variable were selected as the reference level. School sampling weights were used. 


SOURCE: 


International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


3.4.2.3 


Bias analysis for final sample with nonresponse adjusted weights 


This analysis evaluated the potential bias due to nonresponding schools, after 
accounting for the use of replacement schools and nonresponse weight adjustments. 
The analyses conducted were the same as for the final sample before nonresponse 
adjustment (section 3.4.2.2), except that the participating sample was weighted by the 
final school analysis weights, which have been adjusted for nonresponse for the 
purpose of reducing nonresponse bias. The logistic regression conducted in the 
previous analyses is not practical using analysis weights because the weights for the 


nonresponding schools are zero. 


The evidence of nonresponse bias prior to nonresponse adjustment was small for 
many variables prior to the nonresponse adjustment; hence, there was little change in 
the results for these variables. Most estimates of bias, relative bias, and effect size 
(shown in tables 3-15 through 3-18) were similar to or smaller than the estimates in 
the bias analysis for the final sample prior to nonresponse adjustment (tables 3-9 
through 3-12); a few increased slightly in magnitude but were still quite small and not 
statistically significant. All effect sizes for differences between the eligible and 
participating schools were less than 0.05, which is generally considered quite small 
(Cohen 1988). 
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The only statistically significant result is for the chi-square test of independence for 
response status and Census region (table 3-15). The South and Northeast Census 
regions had slightly higher representation among the participating schools compared 
to eligible schools overall, while the Midwest and West Census regions had slightly 
lower representation. The category-level bias estimates and effect size for Census 
region were still smaller compared to the analysis for the final sample before 
nonresponse adjustment (table 3-9). The estimated bias, relative bias, and effect size 


were all smaller than they were prior to nonresponse bias adjustment. 


Table 3-15. Percentage distribution of eligible and participating schools in the U.S. PIRLS 
nonresponse-adjusted sample, by selected categorical variables and nonresponse 
bias indicators: 2016 


Sample schools 


Eligible Participating Relative Chi-square Effect 
School characteristic (percent) (percent) Bias bias p-value size 
School control 
Public 80.15 79.06 -1.09 -0.01 0.3407 0.03 
Private 19.85 20.94 1.09 0.06 
Locale 
City 35.13 36.09 0.96 0.03 0.5425 0.04 
Suburb 32.99 31.74 -1.26 -0.04 
Town 13.72 14.65 0.93 0.07 
Rural 18.15 17.52 -0.63 -0.03 
Census region 
Northeast 15.48 15.93 0.45 0.03 0.0097 0.02 
Midwest 26.34 25.94 -0.39 -0.01 
South 37.94 38.65 0.71 0.02 
West 20.24 19.47 -0.77 -0.04 
Poverty level 
High 72.78 72.28 -0.49 -0.01 0.8845 0.01 
Low 27.22 27.72 0.49 0.02 


NOTE: A high-poverty public school is defined as one in which 76 percent or more of the students are eligible for participation in the free or 
reduced-price lunch program, and low poverty is defined as having less than 76 percent eligible. All private schools are treated as low- 
poverty schools. For definitions of Locale, see https://nces.ed.gov/surveys/urbaned/definitions.asp. For definitions of Census regions, see 
https://www2.census.gov/geo/docs/maps-data/maps/req_div.txt. Eligible schools are those with any fourth-grade students. Participating 
schools are those in which students were assessed. Bias is estimated as the difference between the participant and the eligible sample 
estimates. The relative bias is calculated as the estimated bias divided by the eligible sample estimate. Effect size is calculated as the square 
root of the sum over categories of the squared bias estimates over full-sample means. The chi-square p-value is the result of a test for 
independence between participation status and the variable indicated. All calculations use school sampling weights for the full eligible sample 
and nonresponse-adjusted weights for the participating sample. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Table 3-16. Mean enrollment of eligible and participating schools in the U.S. PIRLS nonresponse- 
adjusted sample and nonresponse bias indicators: 2016 


Sample schools 


Eligible Participating Relative ttest Effect 
Enrollment level (mean) (mean) Bias bias p-value size 
Total school 431.63 426.67 4.96 0.01 0.9178 0.02 
Fourth grade 64.16 63.97 0.19 0.00 0.9824 0.00 


NOTE: Eligible schools are those with any fourth-grade students. Participating schools are those in which students were assessed. Bias is 
estimated as the difference between the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias 
divided by the eligible sample estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value 
reported is the result of the significance test for the difference between bias and zero. All calculations use school sampling weights for the full 
eligible sample and nonresponse-adjusted weights for the participating sample. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


Table 3-17. Mean percentage of students in eligible and participating schools in the U.S. PIRLS 
nonresponse-adjusted sample and nonresponse bias indicators, by race/ethnicity: 


2016 
Sample schools 

Eligible Participating Relative ttest Effect 
Race/ethnicity (percent) (percent) Bias bias p-value size 
White, non-Hispanic 51.58 51.65 -0.07 -0.00 0.9921 0.00 
Black, non-Hispanic 17.05 17.45 -0.40 -0.02 0.9478 0.02 
Hispanic 19.37 18.84 0.53 0.03 0.8893 0.02 
Asian 4.10 3.70 0.40 0.10 0.7815 0.05 
American Indian or Alaska Native 1.87 2.23 -0.36 -0.19 0.8444 0.04 
Native Hawaiian or Pacific Islander 0.18 0.18 -0.00 -0.02 0.9682 0.01 
Two or more races 3.88 3.81 0.07 0.02 0.9429 0.02 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude Hispanic origin. Eligible schools are those 
with any fourth-grade students. Participating schools are those in which students were assessed. Bias is estimated as the difference between 
the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias divided by the eligible sample 
estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value reported is the result of the 
significance test for the difference between bias and zero. All calculations use school sampling weights for the full eligible sample and 
nonresponse-adjusted weights for the participating sample. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


Table 3-18. Mean percentage of students eligible for free or reduced-price lunch, in eligible and 
participating public schools in the U.S. PIRLS nonresponse-adjusted sample and 
nonresponse bias indicators: 2016 


Sample schools 


Eligible Participating Relative ttest Effect 

School characteristic (percent) (percent) Bias bias p-value size 
Percentage of students eligible for free or 

reduced-price lunch 58.66 59.59 -0.92 -0.02 0.8465 0.03 


NOTE: Eligible schools are those with any fourth-grade students. Participating schools are those in which students were assessed. Bias is 
estimated as the difference between the participant and the eligible sample estimates. The relative bias is calculated as the estimated bias 
divided by the eligible sample estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value 
reported is the result of the significance test for the difference between bias and zero. All calculations use school sampling weights for the full 
eligible sample and nonresponse-adjusted weights for the participating sample. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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3.4.3 Item Nonresponse Bias 


Previous iterations of PIRLS did not require item nonresponse bias analysis because 
all items had weighted response rates over 85 percent. For PIRLS 2016, one school 
questionnaire item, indicating the presence of someone to assist with homework if a 
dedicated place for homework was present (ACBGO8B), had a response rate of 84.9 
percent; hence, a nonresponse bias analysis was required for this item. Although 
there were 158 participating schools, only 150 returned the school questionnaire. The 
eight schools that participated but did not complete the survey ate considered 
nonrespondents for all items. Because ACBGO8B was not defined for the 71 schools 
that responded that they did not have a dedicated space for homework (ACBG08A), 
the questionnaire nonrespondents comprised 8 of the 11 item nonrespondents. The 
remaining three nonrespondents responded to only a few questions, so that 
nonresponse for this item was due largely to questionnaire nonresponse rather than 


the nature of this particular question. 


The procedures used for this bias analysis were the same as those described above 
for the school nonresponse bias analysis. The analysis was conducted on the final 
sample with replacement schools using school base weights. Analysis weights were 
not used because item nonresponse bias analyses included unit nonrespondents, in 
accordance with NCES standards. None of the categorical frame variables had 
statistically significant relationships with item response (table 3-19). For the 
continuous variables (tables 3-20 to 3-22), bias estimates for percentage of Black 
non-Hispanic and percentage of Asian students were statistically significant. 
Compared to all eligible schools, participating schools had, on average, slightly higher 
percentage enrollment of Black non-Hispanic students and slightly lower percentage 


enrollment of Asian students. 
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Table 3-19. Percentage distribution of schools eligible for and with responses for ACBGO8B in the 
U.S. PIRLS final sample and nonresponse bias indicators, by selected school 
characteristics: 2016 


Sample schools 


Eligible Responding Relative Chi-square Effect 
School characteristic (percent) (percent) Bias bias p-value size 
School control 
Public 75.09 75.57 0.47 0.01 0.9161 0.01 
Private 24.91 24.43 -0.47 -0.02 
Locale 
City 46.03 50.59 4.57 0.10 0.3829 0.14 
Suburb 25.86 21.73 -4.13 -0.16 
Town 11.97 14.16 2.19 0.18 
Rural 16.15 13.52 -2.63 -0.16 
Census region 
Northeast 13.31 12.32 -0.99 -0.07 0.1185 0.19 
Midwest 25.01 19.72 -5.28 -0.21 
South 40.67 49.56 8.89 0.22 
West 21.01 18.40 -2.62 -0.12 
Poverty level 
High 70.18 68.52 -1.66 -0.02 0.6260 0.04 
Low 29.82 31.48 1.66 0.06 


NOTE: A high-poverty public school is defined as one in which 76 percent or more of the students are eligible for participation in the free or 
reduced-price lunch program, and low poverty is defined as having less than 76 percent eligible. All private schools are treated as low- 
poverty schools. For definitions of Locale, see https://nces.ed.gov/surveys/urbaned/definitions.asp. For definitions of Census regions, see 
https://www2.census.gov/qeo/docs/maps-data/maps/req_div.txt. Eligible schools are those with any fourth-grade students. Responding 
schools are those in which students were assessed. Bias is estimated as the difference between the respondent and the eligible sample 
estimates. The relative bias is calculated as the estimated bias divided by the eligible sample estimate. Effect size is calculated as the square 
root of the sum over categories of the squared bias estimates over full-sample means. The chi-square p-value is the result of a test for 
independence between respondent status and the variable indicated. All calculations use school sampling weights. 

SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


Table 3-20. Mean enrollment of schools eligible for and with responses for ACBGO8B in the U.S. 
PIRLS final sample and nonresponse bias indicators: 2016 


Sample schools 


Eligible Responding Relative ttest Effect 
Enrollment level (mean) (mean) Bias bias p-value size 
Total school 426.49 436.03 -9.54 -0.02 0.5952 0.04 
Fourth grade 60.95 62.34 -1.39 -0.02 0.6563 0.03 


NOTE: Eligible schools are those with any fourth-grade students. Responding schools are those that responded to the item. Bias is 
estimated as the difference between the respondent and the eligible sample estimates. The relative bias is calculated as the estimated bias 
divided by the eligible sample estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value 
reported is the result of the significance test for the difference between bias and zero. All calculations use school sampling weights. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Table 3-21. Mean percentage of students in schools eligible for and with responses for ACBGO8B in 
the U.S. PIRLS final sample and nonresponse bias indicators, by race/ethnicity: 2016 


Sample schools 


Eligible Responding Relative ttest Effect 
Race/ethnicity (percent) (percent) Bias bias p-value size 
White, non-Hispanic 43.79 41.62 2.17 0.05 0.4233 0.07 
Black, non-Hispanic 19.94 23.85 -3.91 -0.20 0.0273 0.14 
Hispanic 23.57 24.57 -1.00 -0.04 0.5754 0.04 
Asian 3.90 2.37 1.53 0.39 0.0431 0.19 
American Indian or Alaska Native 2.56 2.60 -0.04 -0.01 0.9523 0.00 
Native Hawaiian or Pacific Islander 0.21 0.24 -0.02 -0.11 0.3650 0.04 
Two or more races 3.50 2.57 0.93 0.27 0.1223 0.22 


NOTE: Black includes African American, and Hispanic includes Latino. Racial categories exclude Hispanic origin. Eligible schools are those 
with any fourth-grade students. Responding schools are those that responded to the item. Bias is estimated as the difference between the 
respondent and eligible sample estimates. The relative bias is calculated as the estimated bias divided by the eligible sample estimate. Effect 
size is calculated as estimated bias divided by estimated standard deviation. The p-value reported is the result of the significance test for the 


difference between bias and zero. All calculations use school sampling weights. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


Table 3-22. Mean percentage of students eligible for free and reduced price lunch in schools 
eligible for and with responses for ACBGO8B in the U.S. PIRLS final sample and 
nonresponse bias indicators: 2016 

Sample schools 


Eligible Responding Relative ttest Effect 
School characteristic (percent) (percent) Bias bias p-value size 


Percentage of students eligible for free or 
reduced-price lunch 63.49 65.82 -2.33 -0.04 0.2097 0.06 


NOTE: Eligible schools are those with any fourth-grade students. Responding schools are those that responded to the item. Bias is 
estimated as the difference between the respondent and the eligible sample estimates. The relative bias is calculated as the estimated bias 
divided by the eligible sample estimate. Effect size is calculated as estimated bias divided by estimated standard deviation. The p-value 
reported is the result of the significance test for the difference between bias and zero. All calculations use school sampling weights. 
SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


3.4.4 Summary 


The school-level bias analysis found that some statistically significant bias estimates 
remained after nonresponse adjustment; however, the magnitudes of these bias 
estimates were quite small. Nonetheless, analysts may want to take these results into 
consideration when analyzing the data. Overall, the magnitudes of the estimated 
differences between the participating schools and the full eligible sample were smaller 
when replacement schools and nonresponse weight adjustments were used. One survey 
item had a response rate below 85 percent and required a nonresponse bias analysis. 
This analysis found statistically significant differences between responding and 
nonresponding schools for two frame variables. The extent to which the results of these 
bias analyses can be applied to analyses of survey items depend on the extent to which 


the items are correlated with the frame variables used in these analyses. 
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4.1.1 


Survey Operations 


This chapter describes data collection and related activities for PIRLS 2016 in the 
United States. These activities included recruitment of schools, sampling of students 
within schools, development of the instruments used, field operations undertaken to 
administer the assessment, postassessment activities associated with scoring and data 
entry, and several activities associated with the preparation of the data to meet 


international standards. 


Recruiting Districts and Schools 


The established protocol for seeking the participation of schools in studies such as 
PIRLS, where participation is voluntary, is to (1) notify state education authorities of 
the intention to approach schools within their jurisdiction, (2) inform authorities at 
the district level that schools within their districts are being sampled, and (3) contact 
the sampled schools. Participation may be refused at any of these levels, so several 
considerations were important in this context, specifically the need to establish the 
value of participation; establish the timing of the assessment window in conjunction 
with mandatory federal, state, and local assessments; and address concerns about the 
burden on schools. In the case of Catholic schools, the diocese was informed, and 
schools were then contacted. Nonreligious affiliated private schools were contacted 


directly. 


Recruitment activities spanned April 2015 through May 2016, as shown in exhibit 1-1 
in chapter 1. 


The recruitment for PIRLS 2016 benefited from increased involvement at the state 
level. NAEP state coordinators within each state helped to secure participation of 
sampled public schools in their states. 


Contacting States 


Schools from 48 states were sampled for PIRLS. The chief state school officer and 
state assessment director in each of the 48 states with at least one school sampled for 
PIRLS were contacted beginning in April 2015. State contacting materials included a 


letter from NCES, a study brochure, and PIRLS Frequently Asked Questions (FAQ). 


A copy of these materials is provided in appendix A. 
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A webinar was conducted for NAEP state coordinators in each state to explain the 
study and enlist their help to secure school participation. These NAEP state 
coordinators helped to secure support from their state superintendent in many 


instances. 


4.1.2 Contacting Districts 


After the states were informed, similar packages of advance materials were sent to 
the superintendent of each district (or diocese) containing sampled schools. A copy 
of the letter sent to districts is provided as exhibit A-2 in appendix A. 


If the NAEP state coordinator was able to secure support from the state, the NAEP 
coordinator made this contact and included a letter from the state encouraging 


district and school participation in the study. 


In each case, field staff made follow-up calls to district contacts after a few days to 
discuss the study and answer questions. Four school districts required a formal 
application process and school board approval. Several school districts had specific 
security requirements, such as a background check and fingerprinting, in addition to 
the Position of Trust clearance and fingerprints required to work on NCES studies. 
Where this was required, RTI complied with all district-level security requirements. 


During the recruitment period, if a school district or a sampled school in a 
cooperating district refused to participate and was judged to be a firm refusal, a 
similar district package was sent out to the district of the first substitute school 
linked to the sampled school. A parallel procedure was adopted with the second 
substitute district and school in those cases where a first substitute school refused to 


participate. 


4.1.3 Contacting Schools 


After district approval was secured, schools were contacted with an initial school 
information packet. Private schools and some parochial schools not linked with a 
diocese were contacted directly. Because district approvals were received on a flow 
basis, the school recruitment package was also sent on a flow basis after district 
approval was received. A copy of the school recruitment letter is shown as exhibit A-3 


in appendix A. 


After a few days, a member of RTI’s recruitment team contacted the school to 
discuss the school’s participation in PIRLS. In-person visits were made in a small 


number of schools where efforts to secure participation proved difficult. 
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Nomination of school coordinator. School principals were asked to identify an 
individual within the school who would act as the PIRLS school coordinator. 
Principals, assistant principals, teachers, and guidance counselors typically served in 
this role. School coordinator responsibilities are described in exhibit A-5 of appendix 
A and included: 


@ providing a list of all fourth-grade classrooms using the Class Listing Form 
and identifying classrooms to be excluded (see exhibit 4-1); 


e@ completing the Student Listing Form for the selected class(es) and 
identifying any student exclusions (see exhibit 4-2); 


e@ coordinating the date, time, and location of the student session; 
e determining the type of permission form to be distributed to students 
(implicit vs explicit permission), shown in exhibits A-8 and A-9 of 


appendix A; 


e distributing permission forms to students in selected classrooms and 


monitoring the return of signed forms from parents; 


e coordinating the distribution and completion of the administrator and 


teacher questionnaires (Appendix C); 
@ notifying and reminding teachers and students about the sessions; and 


@ maintaining any confidential files for one year and then destroying them. 


4.1.4 Respondent Incentives 


4.1.5 


After the assessment, schools and school coordinators were paid incentives of $200 
and $100, respectively. School coordinators received an additional $50 for assisting 
with ePIRLS. Students each received a plastic watch. Participating teachers recetved 
$20. 


Difficulties in Gaining Cooperation 


Gaining cooperation for voluntary studies has become increasingly challenging. 
Districts and schools had common reasons for refusing to participate, which 
included 


@ too many other high-stakes assessments for the students; 
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4.1.6 


4.2 


4.2.1 


e loss of instructional time; 
e@ burden on school staff and students; and 
e limited benefit for students and schools. 


To increase participation among original sample schools, a subset of refusing schools 
were offered an $800 incentive rather than the $200 incentive. Of the 19 schools 
offered the increased incentive, 11 schools agreed to participate for the increased 
incentive. Five schools agreed to participate in PIRLS but declined to participate in 
ePIRLS because it added an additional day of data collection. 


Monitoring the Recruiting Progress 


RTI used its School Contacting System (SCS) to document and monitor recruitment 
progress. All contacts with states, school districts, and schools are documented in the 
SCS. Recruiters updated status codes for each school district and school to indicate 
whether the school was participating, refusing, or if recruitment was still in progress. 
Reports were generated daily to monitor recruitment progress. NAEP state 
coordinators used a special template similar to the SCS to monitor the recruitment 


status of public schools in their states. 


Sampling Students Within Schools 


As previously mentioned, school coordinators were asked to provide a list of all 
fourth-grade classrooms and, once classrooms were selected, provide a list of 
students in each classroom. One to two classrooms from each participating school 
were selected to participate in PIRLS. This two-stage process was used to sample the 
students to participate in PIRLS. 


RTI used software developed by the IEA Data Processing Center to sample 
classrooms. This software, WinW3S (EA DPC 2015), provided forms generation, 
data entry, class sampling, student sampling, student-teacher linkages, random 
assignment of assessment booklets to students, production of various survey tracking 
forms, and printing of labels for test instruments and questionnaires. Each 


participating country used WinW3S to ensure standardization of these procedures. 


Obtaining Class Lists from Schools and Class Sampling 


Each school was asked to provide a list of fourth-grade classrooms at the school, 


including the number of students in the class and the teacher’s name. A Class Listing 
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Form (CLF) was provided for this purpose as shown in exhibit 4-1. For most 
schools, it reduced the burden and expedited the process by having the school 
coordinator provide the class listing information to the RTI recruiter by telephone in 
lieu of completing and submitting the form. The RTI recruiter entered the 
information into WinW3S in the order in which the classes were reported on the 


phone. 


During sampling, StatsCanada designated each school to have either one or two 
classes selected based on the expected enrollment at the school. This information 
was populated in WinW3S prior to the software being recetved by RTI. WinW3S 
generated an equal-probability sample of one to two classes (or pseudo-classes) 


whenever possible based on the information in the CLF. 
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Exhibit 4-1. Example of PIRLS fourth-grade Class Listing Form with mock data 


PIRLS 2016 - Class Listing Form 


Participant Country United States 
School Name Readalot Elementary School 
School ID 1234 
School Coordinator Name Mrs. Trunchbu11 


Phone Number 9195551212 


oe 2) 3) 8 5) @ 


Name of Class Teacher or 
Class Name Reading / Language Teacher 


Class Group (column 3): 

Class groups occur when students are assigned to specific classes based on their ability/prior achievement. If applicable in your country, the national center defines the groups 
and codes to be used to identify them. If applicable, further instructions of codes to be used can be found in the School Coordinator Manual. Leave blank, if not applicable. 
Class Exclusion Status (column 5): 

As arule, all classes are to be included. Examples of class-level exclusions include classes where all students belong to at least one of the follow ing three exclusion status 
categories: 1 = students with functional disabilities; 2 = students w ith intellectual disabilities; 3 = non-native language speakers. ff all students in the excluded class do not 
belong to the same exclusion category, please identify the category corresponding to the majority of students. All class-level exclusions must be approved by the national 
center. 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study (PIRLS) 2016. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


4. SURVEY OPERATIONS | 47 | 


4.2.2 Identifying Students and Their Teachers 


Once classes were sampled for PIRLS, the Student Listing Form (SLF) was created 
by WinW23S for each sampled class and output in Excel format. An example SLF is 
provided in exhibit 4-2. The top portion of the form was prefilled by WinW3S based 
on the information entered from the CLF, and rows were generated to reflect the 
number of students reported to be in the class. Three extra records were included at 
the end of the list to account for any student newly enrolled since the CLF 
information was received. The remainder of the form was to be completed by the 
school for each student enrolled in the class. For most schools, the form was 
uploaded onto the secure study website where the school coordinator could login, 


retrieve it, and complete the information electronically. 
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Exhibit 4-2. Example of PIRLS Student Listing Form with mock data 


PIRLS 2016 - Student Listing Form 


Participant Country United States 
School Name Readalot Elementary School 
School ID 1234 
Class Name Mrs. Phelps 
Teacher Name Mrs. Phelps 


oe 2) 3) 8 5) 


Date of 


Student Name 
(First Name, Last Name) 


Fern Arable 1 08 | 29 
Hermione Granger 8 P06 P10 P2005 | 
6 Fo | 


Sequence 
Number 


N 
lo) 
oO 
uw 


wn 


tuart Little 
amona Quimby 
arry Potter 
eruca Salt 
aura Wilder 


a 


fay 
w 
BR 
Ww 


aS 
- 
» 


ay 
uw 
BR 
uw 


er 
wlolN|o 
Piel l Re 
WOlOIN|D 


NI 
Oo 
N 
oO 


N 
pm 
N 
pa 


N 
N 
N 
N 


Gender (column 4): 

1 =Female; 2 = Male 

Exclusion Status (column 5): 

1 = Students with functional disabilities; 2 = Students w ith intellectual disabilities; 3 = Non-native language speakers 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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4.2.3 Student Tracking Form (STF) 


The information provided on the SLF was entered into WinW3S. Two student 
tracking forms (STFs) were then generated for each class, one for PIRLS (exhibit 4- 
3) and one for ePIRLS (exhibit 4-4). WinW3S automatically populated each student’s 
name, ID, date of birth, sex, and booklet number. It also marked if the student was 
part of the reliability sample, which indicated that the student’s assessment would be 
double scored to assess reliability (see section 4.5.2.2). A copy of the form was sent 
to the school coordinator to track parental permission status and to the test 


administrator to distribute the assessment materials and track participation status. 


4.2.4 Teacher Tracking Form (TTF) 


WinW3S also generated a Teacher Tracking Form (ITF). This form enabled test 
administrators to record the participation of teachers and to ensure that each teacher 
received the correct teacher questionnaire. An example of a TTF containing fictitious 


information is provided in exhibit 4-5. 
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Exhibit 4-3. Example of PIRLS Student Tracking Form with mock data 


PIRLS 2016 - Student Tracking Form 


School Name: | Participant Country: 
[a] [b] [c] [d] [e] 


Readalot Elementary 
School 


Language 
Tol Cote) aD) (ol FV-t-an of Test 


3) 3) 


Participation Status 


Main Make-Up 
Session Session 


Date of Birth 
uestionnaire 


ession 


Student Name | Student 
or Number | ID 


ern Arable 12345601 


Reliability Scoring 


Exclusion Status 
PIRLS Booklet 
Booklet 


Questionnaire 


Achievement 
Session 


Achievement 
Session 


Gender 
Session 


| 


iy 


Charlie Bucket * 12345602 | 01 
Hermione Granger : 12345603 
Greg Heffley * 12345604 | 11 


ercy Jackson ’ 1234560 
tuart Little * 1234560 
Ramona Quimby ' 12345607 
arry Potter : 12345608 | 0 

: 12345609 /11 


laura Wilder 12345610 [itfo7f2oosf 1 | 73 | | | | | 
Ce ie 
Ce i 
Cer i 


Gender (column 4): 1 = Female; 2 = Male 

Exclusion Status (column 5): 1 = Students with functional disabilities; 

2 = Students with intellectual disabilities; 3 = Non-native language speakers 
Participation Status (column 8): C = Participated; SA = Participated w ith special 
accommodation; 

A = Absent; P= No parental permission; NA = Left school permanently 


Wn!) U0 
uw 


a 


0 


</=x 
oO 

= 

iS 

fe) 

» 

Nn 

<3) 

—_ 
ct 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Exhibit 4-4. Example of ePIRLS Student Tracking Form with mock data 


ePIRLS 2016 - Student Tracking Form 


School Name: ePIRLS Participant Country: 
[a] [b] [c] [d] [e] 


Readalot Elementary School 


Language 
School ID (ol Fm fp) Class Name|Grade of Test 


3] 5] 7] (3) 
Participation 
Status 


cS) 


Date of Birth 


Student Name Student 


ID 


Exclusion Status 


ePIRLS Task 
Assignment 
Main Session 


Gender 


| 


Session Number 


Session 


MM nim aiinminmas if 


Gender (column 5): 1 = Female; 2 = Male 

Exclusion Status (column 6): 1 = Students with functional disabilities; 

2 = Students with intellectual disabilities; 3 = Non-native language speakers 

ePIRLS Task Assignment (column 7): Identifies the two ePIRLS tasks assigned to each student 
Participation Status (column 8): C= Participated; SA = Participated w ith special accommodation; 

A = Absent; P = No parental permission; NA = Left school permanently; F = Equipment failure during the 
testing session; B = Incompatible or failing equipment before the testing session began; U = USB lost or 
upload failed after the testing session 

Session Number (column 9): Session number, if ePIRLS is administered in more than one session due 
to the number of computers available 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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Exhibit 4-5. Example of PIRLS Teacher Tracking Form with mock data 


Pagel 


PIRLS 2016 - Teacher Tracking Form 


School Name: Participant Country: 


United States 


Readalot Elementary School 


Number of Questionnaire 
Eligible Return 
Teacher Name TeacherID |Link Number Students 


Questionnaire Return Status (column 7): 
N = Not returned; P = Returned paper; O = Returned online 


Use additional sheets TIMSS & PIRLS 


PIRLS 2016 - Teacher Tracking Form if necessary Intemational Study Center 


SOURCE: International Association for the Evaluation of Educational Achievement (IEA), Progress in International Reading Literacy Study (PIRLS) 2016. 
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PIRLS Instruments 


All PIRLS instruments were developed by the IEA as a cooperative effort involving 
representatives from every country participating in the study. The instruments 
included a reading assessment, a student questionnaire, a school questionnaire, and 
teacher questionnaires to be completed by the teachers teaching reading to the 
students in the sampled classrooms. Most countries also participated in a home 
questionnaire that was completed by parents. The home questionnaire was not 
administered in the United States. The ePIRLS instruments included an online 


reading assessment and a brief student questionnaire. 


PIRLS. The assessment for PIRLS consisted of 15 booklets and one reader. The 
reader was a glossy, magazine-like booklet that contained the reading passages with 
the questions in a separate booklet. Some items must remain confidential for use in 
coming PIRLS assessments while others have been released as restricted-use 
passages and items by the IEA on the TIMSS & PIRLS International Study Center 
website, at http://timssandpirls.bc.edu/pitls2016/international-results 


content/uploads/structure/PIRLS/11.-appendices/H_restricted-use-passages- 


questions-and-scoring-guides.pdf. The student questionnaire was bound separately 
from the assessment items. For each student, the assessment booklet and 
questionnaire were labeled with the student ID and placed in a 9x12 envelope that 
also contained the student ID to ensure that each student completed the correct 
booklets. To ensure confidentiality, no student names or school names were included 


on the booklet labels or the student envelope. 


The school and teacher questionnaires were designed as online instruments to be 
completed by the school principal and teacher, respectively. The international 
versions of the PIRLS questionnaires are available on the TIMSS & PIRLS 
International Study Center website at 

https://timssandpirls.bc.edu/pitrls2016 /questionnaires/index.html. The U.S. version 


of the PIRLS questionnaires are provided in appendix C of this report and are 
available on the NCES website https://nces.ed.gov/surveys /pirls/questionnaire.asp. 


ePIRLS. The ePIRLS assessment consisted of five tasks designed to measure how 
well students read, comprehend, and interpret online information. An infographic 
describing the ePIRLS assessment environment is shown in exhibit 4-6. Each 
student completed two tasks, designed to take about 40 minutes each. Students were 
assigned to the two tasks based on a rotation that was designated by the study 
software and built into the student login information. The online assessment was 


followed by a brief electronic questionnaire. ePIRLS was administered by plugging a 
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USB into the computer and launching the ePIRLS system. School computers were 
used when possible, or RTI test administrators brought laptop computers into the 


schools for this administration. 


Exhibit 4-6. ePIRLS assessment screen 


Assessment Window Guides students 
through the ePIRLS assignment and captures 
student data, such as navigation, responses, 

and timestamps. 


Internet Browser Window Students search 
for, navigate through, and read content on multiple 
websites in a simulated Internet environment. 


PIRLS 


My @E , 

) ate earration a createretenture/ pole bee! | ty UbaSsloOM) 

|g Ai F : answer the following question: 
Creature Features : 
~ 9 What is the most important 

| information about polar bears 

1 


ROAD 
IRLP 
4 


feed tb go cul on the sen ke. 
‘They otter teat eertty at a nats 
Drea hole he ee wang 
er 8. S0el me Welker © sutece, 
Once he seul comes up, Fe 

Dear wa png and capt 

Tere 


» 
Gary Paulsen 
a 
Jin Paulsen 


Teacher Now, please go to Panels 2 
and 3 of Polar Bear Facts & Photos and 
fead the information to answer the 

following questions: 


Why do polar bears depend 
on sea ice for their survival? 
Student 


that is where the seals ve and 
polar bears eat seals 


Online Reading Non-linear Online Teacher Avatar 


Passages By visiting Navigation Multiple Sets the context for Responses 
various text panels and tabs and hyperlinks the assessment task, Students answer 
pages, students read provide an authentic prompts students with multiple choice and 
content that helps them online reading experience, questions, and keeps the constructed response 
complete their online supplemented by photos student on track during questions about the 


assignment. and graphics. the assessment. online texts. 


SOURCE: https://timss.bc.edu/pirls2016/downloads/P2016 ePIRLS Brochure.pdf. 
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PIRLS Reading Assessment 


The following discussion provides a summary of the rationale for and development 
of the PIRLS 2016 assessment. Complete detail is provided in the Methods and 
Procedures in PIRLS 2016 (Martin, Mullis, and Hooper 2017), available at 
https://timssandpirls.bc.edu/publications /pirls/2016-methods.html. 


Assessment frameworks 


The PIRLS 2016 test development effort began with a revision of the frameworks 
used to guide the construction of the previous PIRLS 2011 assessments (Mullis et al. 
2009). The frameworks were updated to reflect changes in the curriculum and 
instruction of participating countries (Mullis and Martin 2015). Extensive input from 
experts in reading education, assessment, and curriculum, and representatives from 
national educational centers around the world contributed to the final shape of the 
frameworks. Maintaining the ability to measure change over time was an important 


factor in revising the frameworks. 


Content and cognitive domains 


PIRLS 2016 assessed “purposes of reading” in two content domains, /terary experience 
and acquire and use information. Each of the five ePIRLS tasks were designed to assess 
how students acquire and use information in an online environment. ePIRLS did not 
include a “/ierary experience domain. Both PIRLS and ePIRLS assessed students’ 
reading literacy “processes of comprehension” in four cognitive areas: (1) focus on 
and retrieve explicitly stated information; (2) make straightforward inferences; 

(3) interpret and integrate ideas and information; and (4) examine and evaluate 


content and textual elements. 


Item development 


Approximately half of the assessment items used in PIRLS 2011 were kept 
confidential and included in the PIRLS 2016 assessments. To replace assessment 
items that had been released following the 2011 assessments, education systems 
submitted items for review by subject-matter specialists, and additional items were 
written by the IEA Reading Review Committee for PIRLS, in consultation with 
item-writing specialists. This expert consultation ensured that the content, as 
explicated in the frameworks, was covered adequately. Items were reviewed by the 
Reading Review Committee and field-tested in most of the participating education 
systems. Results from the field test were used to evaluate item difficulty, how well 
items discriminated between high- and low-performing students, the effectiveness of 


distracters in multiple-choice items, scoring suitability and reliability for constructed- 
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response items, and evidence of bias toward or against individual countries or in 


favor of boys or girls. 


As a result of this review in PIRLS, 117 new reading items were selected for 
inclusion in the international assessment. In total, 223 reading items were included in 
the PIRLS 2016 assessment booklets. 


ePIRLS was administered for the first time in 2016. Experts from the TIMSS and 
PIRLS International Study Center at Boston College developed the ePIRLS content 
based on the TIMSS 2015 Science Framework. Storyboards for the ePIRLS tasks 
were thoroughly reviewed by each participating country’s National Research 
Coordinator and the Reading Review Committee. The systems to deliver ePIRLS 
were developed by the IEA Data Processing Center in Hamburg, Germany. Six tasks 
were developed that included simulated internet web pages with multiple pages of 
text. A total of 112 items were included in ePIRLS. 


Mote detail on the distribution of new and trend items in the PIRLS 2016 
assessments and the development of the ePIRLS items can be found in PIRLS 2016 
Assessment Framework (Mullis and Martin 2015). 


4.3.1.4. Assessment booklets 


4.3.2 


The PIRLS 2016 assessment used a rotated block design to minimize burden on 
students. The PIRLS assessment consisted of 15 booklets and a reader that was 
presented in a magazine-type format, with the questions in a separate booklet. Each 
booklet required 80 minutes of response time. The reading items were assembled 
separately into 10 blocks, or clusters, of items. Each of the 16 booklets (15 booklets 
and a reader) contained one block of literary experience items and one block of 
informational items, and each block occurred twice across the 16 booklets. The 16 
booklets were rotated among students with each participating student responding to 
only 1 booklet. Six of the 10 blocks were included in previous PIRLS assessments. 
The remaining 4 blocks were new for PIRLS 2016. 


Like PIRLS, the ePIRLS assessment used a rotated block design to assign the five 
tasks to students. Each task took about 40 minutes. All the tasks were new to the 


study and were focused on how students acquire and use information. 


Background Questionnaires 


As in prior administrations, PIRLS 2016 included self-administered questionnaires 
for principals, teachers, and students. To create the questionnaires for 2011, the 2011 


version of the PIRLS questionnaires were reviewed extensively by the national 
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research coordinators from the participating countries as well as a Questionnaire 
Item Review Committee (QIRC). Based on this review, the QIRC eliminated or 
revised some questions and added several new ones. Like the assessment items, all 
questionnaire items were field-tested, and the results were reviewed. As a result, 
some of the questionnaire items were revised prior to their inclusion in the final 
questionnaires. The questionnaires requested information to help provide a context 
for the performance scores, focusing on such topics as students’ attitudes and beliefs 
about learning, their habits and homework, and their lives both in and outside of 
school; teachers’ attitudes and beliefs about teaching and learning, teaching 
assignments, class size and organization, instructional practices, and participation in 
professional development activities; and principals’ viewpoints on policy and budget 
responsibilities, curriculum and instruction issues, and student behavior, as well as 
descriptions of the organization of schools and courses. The preferred mode of data 
collection for teachers and school administrators was web-based, but paper versions 
were made available to students and to teachers and school administrators for 
nonresponse follow-up. Results from the student, teacher, and school questionnaires 
are available in PIRLS 2016 International Results in Reading (Mullis et al. 2017). 


In addition to the student questionnaire for PIRLS, students who completed the 
ePIRLS assessment were asked to complete a brief computer-based questionnaire for 
ePIRLS. Questions on the ePIRLS student questionnaire provided a context for 
performance on the ePIRLS tasks and focused on accessibility and use of the 


Internet at home and at school. 


4.3.3 U.S. Adaptations to the Assessment Items and Questionnaires 


Source versions of all instruments (assessment booklets, questionnaires, and 
manuals) were prepared by the IEA in English and translated by the participating 
countries into the primary language or languages of instruction in each country. In 
addition, it was sometimes necessary to adapt the instruments to better fit language 
usage, even in countries that use English as the primary language of instruction. 
Other adaptations to fit national education characteristics were sometimes required 
as well. All adaptations were reviewed and approved by the IEA to ensure they did 


not change the substance or intent of the question or answer choices. 


4.3.3.1 U.S. adaptations to the assessment items 


As in previous cycles of PIRLS, the U.S. adaptations to the international instruments 
were minimal and designed to make the assessment more readable to U.S. students 
without changing the essence of the assessment item. For example, nouns with 


British origins were changed to their U.S. equivalent (for example, “mum” to “mom” 
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or “lift” to “elevator”’), British English spellings were changed to American English 
(for example, “grey” to “gray’’), mathematical terms were changed from the metric 


system to the imperial system (for example, “meters” to “feet’’). 


U.S. adaptations to the school, teacher, and student questionnaires 


Three types of adaptations were made to the school, teacher, and student 


questionnaires: 


e changes designed to make question text more readable to U.S. students, 


similar to those made to the assessment items as described above; 


e@ changes to response alternatives where the international response set did 


not adequately reflect the U.S. context; and 
e@ additional questionnaire items included to address issues of national interest. 


A detailed list of changes made to the questionnaires is provided in appendix D. 
Both the original text from the international version of the questionnaire and the 
changed text from the U.S. version are shown. Text that has been changed in the 
USS. version is underlined in that version. Both international and U.S. questionnaire 
item numbers, or other location indicators, ate provided in each instance. Where 
approptiate, a crosswalk between the U.S. and international versions of the set of 


response categories of items is provided in the “Comments” column. 


Translation and Verification of Instruments 


Each country prepared translations of the instruments according to translation 
guidelines established by the IEA. Since the international versions of the instruments 
are produced in English, the United States did not need to engage in the full-fledged 
translation required of many nations. However, the adaptations made to the U.S. 
instruments required verification by the IEA to ensure their suitability for the current 
cycle of PIRLS and, if trend items, their continuity with previous cycles. Further 
details on the translation process can be found in Martin, Mullis, and Hooper (2017). 


Production of Assessment Booklets and Questionnaires 


On receiving IEA approval of the adaptations, RTI applied the adaptations to the 
international questionnaires and item blocks and then assembled the final assessment 
booklets. Quality control procedures for this process included a review of each 
adaptation made to the questionnaires and item blocks as well as a full review of the 


assembled instruments in a final layout proof. 
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In mid-December 2015, electronic files were sent to the IEA Data Processing Center 
(DPC) for verification of the national changes and to the TIMSS & PIRLS 
International Study Center for layout verifications. The student, teacher, and school 


paper-and-pencil versions of the questionnaires were printed in nonscannable form. 


A similar procedure was applied to the online school and teacher questionnaires. 
Adaptations were made to the online instruments using software provided by the 
IEA DPC. A set of output files was produced for each questionnaire, and these were 
uploaded to a secure sever and verified by the DPC. 


Preparation and printing of paper-and-pencil Instruments 


For all paper and pencil instruments, proofs of each document were reviewed against 
the original electronic files prior to printing. Once accuracy was certified, printing 
was initiated. During this process, staff checked a 10 percent sample of the printed 
form against the approved document to ensure that accuracy was maintained 


throughout the printing process. 


Online questionnaires 


Once verified, each questionnaire was loaded to an NCES server and thoroughly 
tested by RTI to ensure that responses were being captured correctly and that the 


instruments were functioning properly. 


ePIRLS data 


The collection of ePIRLS task and questionnaire data was completed via software on 
USB sticks. Test Administrators (TAs) inserted the USB stick into the school 
computer or study-provided laptop prior to the session and all data were stored on 
the USB stick. The TA then took the USB sticks home with them after the session 
and uploaded the data from each stick to the secure study server. 


Field Operations 


In the United States, PIRLS and ePIRLS were administered by professional staff 
trained according to the international guidelines. School personnel were asked only 
to assist with listings of students, the identification of school space for the 
assessment, and the specification of parental consent procedures needed for sampled 
students. 
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4.4.1 


Forty-nine TAs were hired to work on PIRLS 2016. All had previous experience 
with other educational assessments in schools, and all received Position of Trust 
clearance based on fingerprint and background checks. TAs also signed a statement 
of nondisclosure indicating that they would maintain confidentiality of all survey 
materials and of the data collected. The TAs recruited local test administrator 
assistants ([AAs) to assist in the administration of the assessment because, in most 
schools, two separate classrooms were assessed simultaneously. TAAs were also 
experienced with the administration of educational assessments in schools and 


received a background check. They too signed confidentiality statements. 


The TAs reported to one of four field supervisors who coordinated and monitored 
their work, and, in turn, the TAs coordinated and supervised the work of the TAAs. 
The field supervisors reported to the national field manager at the RTI home office. 


Responsibilities of Field Supervisors, Test Administrators, and Test 
Administrator Assistants 


Field supervisors had responsibility for 
@ participating in the PIRLS data collection training; 
e tracking the TAs’ receipt of assessment booklets and other materials; 
@ coordinating data collection activities undertaken by their assigned TAs; 


e@ holding weekly one-on-one telephone meetings with their TAs to monitor 


progress and to troubleshoot any problems arising; 
e@ ensuring that their TAs followed PIRLS procedures and guidelines; and 


@ reporting progress and problems in weekly conference calls with RTI home 


office staff and other field managers. 
Test administrators had responsibility for 
e@ attending PIRLS data collection training; 
@ receiving and securing assessment materials; 
e training TAAs; 
@ preparing and assigning assessment materials for students; 


e collecting the completed school and teacher questionnaires; 
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e@ conducting the assessment according to PIRLS-specified procedures; 


@ completing the Test Administration Form, Student Tracking Form, Teacher 
Tracking Form, and Student Response Rate Form; 


e determining if a follow-up session was needed; 


@ securing, packing, and shipping all assessment materials to RTI at the 


conclusion of the assessment; 
e recording the status of the assessment in the Field Reporting System; and 
@ reporting progress to their field supervisor on a regular basis. 
Test administrator assistants had responsibility for 
e@ attending the PIRLS training conducted by their TA; 
e@ administering the assessment according to PIRLS-specified procedures; 


@ completing the Test Administration Form, Student Tracking Form, Teacher 
Tracking Form, and Student Response Rate Form; and 


e@ consulting regularly with their TA. 


4.4.2 Training 


A 4-day, in-person training for TAs was held February 1-4, 2016. The attendees 
received the Test Administrator Manual 10 days prior to the training session and 
were given four paid “study hours” to become familiar with the information prior to 
training. The agenda for this training session is provided as exhibit B-1 in 

appendix B. 


The training covered both PIRLS and ePIRLS administration. The first day focused 
on the study background, TA responsibilities, preassessment activities, and data 
security. The second day started with discussion about the procedures to be followed 
in preparing for, arriving on, and conducting the assessment day; what to do once 
the assessment was completed; and the appropriate methods for packing and 
shipping the assessment materials to RTI. The afternoon focused on introducing 
ePIRLS. Day 3 focused on the administration of ePIRLS, completing the study 
forms, and quality control procedures. The final day focused on administrative 
procedures and a recap of the prior days of training, followed by a certification to 


confirm that the TAs were prepared to conduct the session in the schools. 
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4.4.3 


4.4.4 


4.4.4.1 


Field staff were assigned laptop computers to take with them for the duration of 
PIRLS data collection. Test administrators were also provided with an official photo 
ID badge to wear while representing PIRLS in the schools. 


Training for TAAs was conducted as a separate exercise by their respective TAs. 
Approximately 2 hours was allocated for this training. Training materials consisted of 
a Test Administrator Assistant Manual, session scripts, and a subset of material from 
the TA training. 


Assignment of Schools to Test Administrators 


Test administrators were assigned a work area based on their location and 
availability, with most work areas relatively close to the test administrator’s home 
address. Because sampled schools were not geographically clustered, some schools 
required the TA to travel. Balancing these several demands resulted in some variation 
in the caseload of TAs. During the assessment period, which ran from February 
2016 through May 2016, some reassignment of schools and/or work areas was 


necessary. 


Assessment and Related Activities 


TAs engaged in a number of activities before the actual assessment. These included 


e working with the school coordinator to gain the permission of parents and 


students, if this was required by the school; 
e@ making arrangements with the school for the assessment sessions; and 


e@ obtaining the materials to be used in the assessment. 


Recruiting parents and students 


During recruitment and scheduling contacts with schools, recruitment staff asked 
about district and/or school requirements for notifying parents about their child’s 
participation or obtaining parental permission. School requirements fell into the 


following two categories: 


e Implied permission (passive/opt-out). The school was required to ask 
parents for permission for the child to participate, but permission would be 


assumed unless there was a formal objection; and 
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e Explicit permission (active/opt-in). The school was required to ask 
parents for permission for the child to participate, and the child could not 


participate until the parents provided formal written approval. 


A majority of schools (141) opted for implied permission, while 17 required explicit 
permission. Copies of both permission types are shown in exhibit A-8 and 
exhibit A-9. 


4.4.4.2 | Organizing the assessment session at the school 


Approximately 2—4 weeks before each school’s assessment date, the TA called the 
assigned school coordinator. Test administrators were instructed to verify previously 
obtained information on items such as the school’s address, principal’s name, 
assessment date, session location, requirements for entering the school, and parking 
arrangements, as well as the status of the within-school sampling forms for the 
school. The information obtained was updated and maintained in the Field 
Reporting System. 


PIRLS. On assessment day, each TA, accompanied by a TAA, arrived at the school 
about an hour before the scheduled assessment with all of the materials needed for 
the assessment. One session box of materials was provided for each of the sampled 
classes. Each session box contained the estimated number of student assessment 
booklets required, plus three unassigned booklets to accommodate any changes in 
class enrollments. Upon arrival, the TA met with the school coordinator to make any 
updates to the Student Tracking Form that would affect the preparation of student 
materials (for example, the addition of new students, the withdrawal of listed 
students from the school or class, a change in exclusion status of a sampled student, 


or change in permission status). 


ePIRLS. ePIRLS typically took place the day after the PIRLS administration. The 
TA arrived about 90 minutes before the scheduled assessment to set up the 
computers. School computers, computers brought in by RTI, or a combination of 
the two options were used for ePIRLS. The TA arrived with the USB sticks loaded 
with ePIRLS and launched the program on each of the computers before the 
students arrived. Students could only participate in ePIRLS if they had already 
participated in PIRLS. Students who were absent for the first session completed the 
PIRLS paper-and-pencil assessment instead of ePIRLS that day. 


4.4.4.3 Administering the assessment 


Assessments were administered by trained test administrators who tread verbatim 


from a standardized script according to the instructions in the PIRLS Test 
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Administrator Manual. A copy of the session script is provided in appendix B. All 
activities for PIRLS and ePIRLS, including the timing of each component and any 
unusual occurrences, were recorded on the Test Administration Form provided in 


appendix B. 


PIRLS. The script began with a brief introduction to the study. The assessment 
booklets, each in a security envelope, then were distributed. The students were 
instructed to remove their booklet from the envelope, and the general instructions 
and instructions for Part 1 were read. Following this, the students were instructed to 
begin Part 1 of the assessment. After 40 minutes, a short break was provided. After 
the break, the instructions for Part 2 were read and students were instructed to begin 
Part 2 of the assessment. After the allotted 40 minutes for this part of the 
assessment, students were instructed to stop work, and another break was provided. 
Following the break, the student questionnaire was administered; it was not time 


limited but was typically completed in about 30 minutes. 


ePIRLS. The script began by explaining ePIRLS and walked the students through 
the directions for completing the tasks on the computer. Students were then given 40 
minutes to complete each task, with a 5-minute break between tasks. After the 
allotted 40 minutes for the second task, students were given another 5-minute break 


before completing the 5-minute student questionnaire. 


4.4.4.4 Postassessment activities 


45 


Following the assessment, test administrators instructed the students to place the 
booklets back in the 9x12 envelope. The students handed their envelopes to the TA, 
received their gift, and were dismissed. The test administrator then recorded 
participation codes for each session and packed the envelopes containing the student 
booklets and questionnaires and, if applicable, school and teacher questionnaires into 
the shipping box. The TA packed up the materials but kept the Student Tracking 
Form in their possession to key the participation information into the field reporting 
system. The session materials were sealed and shipped to RTI. The STFs were 


shredded when all information was complete for the school. 


Receipt Control, Scoring, Coding, and Data Entry 


As noted previously, field staff sent the completed assessments and questionnaires 
along with any related materials directly to RTI following the completion of the 
assessment session at a school. RTI then recorded the receipt of materials and keyed 


the paper-and-pencil questionnaire data. Assessment data were scored and coded by 
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Measurement Inc. prior to being keyed by RTI. Data files were then created from the 
data. 


Receipt Control 


PIRLS documents were received at RTI between February and May 2016. The 
school, student, teacher, or administrator [Ds for the booklets received were 
recorded in RTT’s receipt control system. Information recorded in the receipt control 
system was systematically compared to the information recorded in the field 
reporting system. A discrepancy report was generated daily and used to reconcile any 
inconsistency between the materials received and the information recorded from the 
student tracking form into the field reporting system. 


When the return shipments were received by RTI, a manual count was made to 
ensure that all booklets from the original shipment were included. The assessment 
booklets were sorted by booklet type and prepared for delivery to Measurement Inc. 
for scoring. Questionnaires were submitted for data entry. Unused booklets were 


batched separately. 


Scoring the Assessment Items 


The PIRLS assessment items included both multiple-choice and constructed- 
response items. Scoring rubrics developed internationally following the field tests of 
the assessment items were available to guide the scoring of each constructed- 
response item. In the United States, the scoring of the open-ended student responses 


according to these rubrics was the responsibility of Measurement Inc. 


Training 


The national research coordinator, the U.S. scoring manager (from Measurement 
Inc., which conducted the scoring), and RTI International quality monitoring staff 
participated in the PIRLS scoring training sessions sponsored by the IEA. Materials 
from these sessions, along with additional materials constructed specifically for this 
purpose, were used to train the scoring director, scoring team leaders, and scoring 
team members. Scorers were hired based on their experience with similar reading 
scoring projects. Measurement Inc. hired 16 scorers for PIRLS, organized into two 
teams, each overseen by a separate scoring team leader. All teams were trained and 


monitored by a separate scoring director and the overall scoring manager. 


Once trained, the scoring director and scoring team leaders each led a small team of 
scorers in reading the item prompt; reading the rubric or scoring guide aloud; reading 


aloud each of the anchor papers and explaining the reasoning behind the score; 
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4.5.2.2 


4.5.2.3 


4.5.2.4 


allowing the scorers time to complete the practice papers; reviewing each of the 
practice papers; and scoring booklets. Training on and scoring of the items was 
completed in batches, with scorers trained on one or two passages before starting to 


score booklets containing those passages. 


Scoring 


Each team worked on all passages and items. Student booklets marked as being part 
of the reliability sample had their assessments scored separately by two different 
scorers. To reduce training fatigue and promote retention of rubric guidelines, 
scorers were trained on one or two passages and worked on scoring those booklets 


before being trained on the next set of passages. 


Scoring quality was monitored continuously. Team leaders, the scoring director, and 


the scoring manager conducted “read-behinds” to monitor the reliability of scoring. 


Cross-country scoring reliability study 


In international assessments, it is also important to gather information about how 
reliably the scoring was conducted from country to country so that valid 
international comparisons can be made of students’ achievement. To document the 
reliability of constructed-response scoring, a cross-country scoring reliability study 
was conducted. Responses to PIRLS items from Southern Hemisphere countries 
were sent to the Northern Hemisphere countries for scoring. After the scoring of the 
US. responses for a passage with their team, scorers then scored the international 


responses that were preloaded on desktop computers. 


Trend scoring 


To document the reliability of constructed-response scoring from PIRLS 2011 to 
PIRLS 2016, PIRLS included a trend scoring reliability study. This study estimated 
the degree of agreement between the 2016 scorers and the scorers from the previous 
assessment. It allowed scorers of the PIRLS 2016 assessment to score student 
responses collected in 2011. The PIRLS 2016 scorers scored a subset of student 
responses from the prior PIRLS instruments. Student responses were actual student 


responses to items collected during the PIRLS assessments in the United States. 


The IEA DPC assembled a sample of student responses to be scored and distributed 
it to each participating education system along with the IEA Trend Scoring 
Reliability Software (SRS). Only education systems participating in 2016 that also 
participated in PIRLS 2011 took part in the trend scoring reliability study. 
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Data Entry of Questionnaire and Assessment Responses 


Student questionnaire and assessment data were entered into a data entry system 
provided by the DPC. For school and teacher questionnaires received as hard copy, 
the questionnaire data were entered directly into the online questionnaire for that 
school or teacher. School and teacher questionnaire data were imported into the 
Data Management Expert, or WinDME, provided by the IEA Data Processing 


Center. 


File Creation and Consistency Checks 


In a final step, the data from the assessment score files were merged with the student 
scanned data. Then final output files were produced for each file type. The final files 
were checked to ensure that the data were completeness, in compliance with 
codebook specifications, and in the correct format. In addition, a check was 
performed to verify correct linking and matching of student, teacher, and school data 
files. Student and teacher files were loaded in the WinDME software so that all data 
from the assessments and questionnaires were available in the format required by the 
IEA. 


Data Preparation 


As noted in the previous section, the data collected for PIRLS 2016 were entered 
into data files according to a common international format, as specified in the 
WinDME data entry software. The software facilitated the checking and correction 


of data by providing various data consistency checks. 


The data files in this format were sent to the IEA DPC, where they were subjected 
to an extensive series of data cleaning and consistency checks. The overriding 
concern of these checks was to ensure that all information in the database 
conformed to the internationally defined data structure, national adaptations to 
questionnaires were reflected appropriately in the codebooks and documentation, 
and all variables used for international comparisons were comparable across 


countries. 


International Data Cleaning Procedures 


The DPC was responsible for checking the data files from each country, applying 
standard cleaning rules to verify the accuracy and consistency of the data, and 
documenting electronically any deviations from the international file structure. 


Queries arising during this process were addressed to national centers, and this 
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process was repeated as necessary to ensure that the data were consistent and 


comparable within and between countries. 


Following this cleaning step, countries were provided national univariate and 
reliability statistics along with data almanacs containing international univariate 
statistics and item statistics. This allowed countries to examine their data with those 
of other participating nations. Once any problems arising from this examination 
were tesolved, sampling weights produced by Statistics Canada and IRT-scaled 


student proficiency scores in mathematics and science were added to the file. 


Detailed information on the entire data entry and cleaning process can be found in 
Martin, Mullis, and Hooper (2017). 


Data Confidentiality Safeguards 


NCES and data contractors routinely pledge confidentiality to respondents; however, 
over the past decade, concerns about the potential for disclosure of information 
about individual survey respondents have increased. These concerns are reflected in 
new laws enacted since the Privacy Act of 1974 to further ensure the protection of 
confidential data. The Education Sciences Reform Act of 2002 explicitly requires 
that NCES protect the confidentiality of all those responding to NCES-sponsored 
sutveys so that no individual respondent can be identified. More specifically, NCES 
standard 4-2, Maintaining Confidentiality (Seastrom 2014), provides guidelines for 
limiting the risk of data disclosure for data released by NCES. Data disclosure occurs 
when an individual respondent has been identified through the use of the survey 
item responses and other external data sources. The procedures used to reduce the 
risk of data disclosure for PIRLS 2016 in accordance with the guidelines specified in 
NCES standard 4-2, are described below. 


All students, teachers, and schools participating in PIRLS 2016 do so with the 
assurance that their identities will not be disclosed. Confidentiality procedures in 


place included the following: 


e All employees with access to the data signed affidavits of data 
confidentiality. 


e No student names were included on the questionnaires or assessment 
booklets. 


In addition to data collected directly from schools, teachers, and students, additional 
information was used during the PIRLS sampling, data collection, and weighting 
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processes; and these variables also were considered as part of the review to 


determine disclosure risk levels. 


The confidentiality analysis review described below used the following three-step 


process to reduce disclosure risk: 
e determining the disclosure risk arising from existing external data; 
@ creating a derived variable for race; and 
@ swapping the data. 


In this process, additional assurance is provided that individual schools, teachers, and 
students participating in PIRLS could not be identified through comparison with 
public data collections once the PIRLS data were released for public use. Although 
no public data collections identify students or teachers by name, three publicly 
available data collections do identify schools by name. These are the CCD, a detailed 
public school listing; the PSS, a detailed private school listing; and the Quality 
Education Data (QED) produced by Market Data Retrieval (MDR), a privately 
owned education research firm. The QED data contain a school-based file that 
provides demographic information for both public and private schools along with 
the names of the schools. Thus, there is some possibility that schools at least, and 
perhaps teachers and students as well, could be identified if comparisons of these 
datasets with the PIRLS dataset allowed the identification of schools. 


Disclosure analyses were conducted to minimize the possibility that schools, teachers 
or students could be identified in the dataset. School matching analyses were 
undertaken using probabilistic matching algorithms approved by the IES Disclosure 
Review Board (DRB) for use in disclosure analyses. These algorithms identify 
schools with some potential for identification. To provide further protection, 
elements of the data from schools identified as “disclosure risks” in this way were 
perturbed using the procedures approved by the DRB. After perturbation, the data 
were subjected to another round of analyses to ensure that the potential for 


identification no longer existed. 


An additional measure was taken to reduce further the risk of disclosure of an 
individual respondent. This measure is referred to as data swapping, a DRB 
requirement that reduces risk by modifying microdata. In data swapping, a 
probability sample of records is paired with other records on the file using selected 
characteristics, and then some identifying variables are swapped between pairs of 
records (see Kaufman, Seastrom, and Roey 2005). The sampling rate for PIRLS 
swapping was designed to protect the confidentiality of the data without affecting the 
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usability of the dataset. All questionnaire data (school, teacher, and student) were 
involved in the swapping. This method is an effective way of keeping as much 
valuable data as possible while protecting respondent identity. Swapping preserves 
the univariate frequencies, means, and variances, although it may affect multivariate 
relationships a little. Pre- and postswapping percentage distributions (unweighted 


and weighted) and correlations were reviewed to ensure data quality was maintained. 


Confidentiality analyses of this kind were conducted before the U.S. data files were 
delivered to the DPC for cleaning and prior to the IRT scaling and estimation of 
sampling weights. 


Estimation of PIRLS Student Proficiencies 


All cycles of PIRLS used IRT methods to produce score scales that summarized the 
achievement results. With this method, the performance of a sample of students in a 
subject atea or subarea could be summarized on a single scale or a series of scales, 


even when different students had been administered different items. 


IRT scaling provides estimates of item parameters (for example, item difficulty and 
item discrimination) that define the relationship between the item and the underlying 
variable measured by the test. Parameters of the IRT model are estimated for each 
test item, with an overall scale being established as well as scales for each content 


area and cognitive domain specified in the assessment framework. 


To allow for the calculation of trends in achievement, comparisons of scores across 
the PIRLS assessments conducted in 2001, 2006, and 2011 were necessary. To this 
end, achievement scores from all three PIRLS cycles were placed on the same scale. 
Details are provided in Methods and Procedures in PIRLS 2016 (Martin, Mullis, and 
Hooper 2017). 


Plausible values 


During the scaling phase, plausible values were used to characterize scale scores for 
students participating in the assessment. To keep student burden to a minimum 
while ensuring content coverage, PIRLS administered a limited number of 
assessment items to each student—too few to produce accurate scale scores for each 
student. To account for this, PIRLS generated five possible scale scores for each 
student, each representing a random selection from the distribution of scale scores of 


students with similar backgrounds who answered the assessment items the same way. 


This plausible-values methodology was used to represent what the true performance 
of an individual might have been, had it been observed. This is done by using a small 
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number of random draws from an empirically derived distribution of score values 
based on (1) the student’s observed responses to assessment items and 

(2) background variables. Each random draw from the distribution is considered a 
representative value from the distribution of potential scale scores for all students in 
the sample who have similar characteristics and identical patterns of item responses. 
The draws from the distribution are different from one another to quantify the 
degree of precision (the width of the spread) in the underlying distribution of 
possible scale scores that could have caused the observed performances. The PIRLS 
plausible values function like point estimates of scale scores for many purposes, but 
they are unlike true point estimates in several respects. They differ from one another 
for any particular student, and the amount of difference quantifies the spread in the 
underlying distribution of possible scale scores for that student. 


This approach to the estimation of scale scores ensures that the estimates of the 
average performance of student populations and the estimates of variability in those 
estimates are more accurate than those determined through traditional procedures, 
which estimate a single score for each student. An accessible treatment of the 
derivation and use of plausible values can be found in Beaton and Gonzalez (1995). 
Details specific to PIRLS can be found in Methods and Procedures in PIRLS 2016 
(Martin, Mullis, and Hooper 2017). 


4.6.3.2 International benchmarks 


International achievement benchmarks were developed to provide a concrete 
interpretation of what the scores on the PIRLS reading scales mean. PIRLS used 
scale anchoring to summarize and describe student achievement at four points on 
the reading scales—Advanced (625), High (550), Intermediate (475), and Low (400). Scale 
anchoring involves selecting benchmarks (scale points) on the PIRLS achievement 
scales to be described in terms of student performance and then identifying items 
that students scoring at the anchor points can answer correctly. Subsequently, these 
items are grouped by content area within benchmarks and reviewed by reading 
experts. These experts focus on the content of each item and describe the kind of 
reading knowledge demonstrated by students answering the item correctly. The 
experts then provide a summary description of performance at each anchor point 
leading to a content-referenced interpretation of the achievement results. Detailed 
information on the creation of the benchmarks is provided in the “Using Scale 
Anchoring to Interpret the PIRLS 2016 Achievement Scales” section of Methods and 
Procedures in PIRLS 2016 (Martin, Mullis, and Hooper 2017). 
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4.6.4 Estimation of Sampling Weights 


Because of the complex sampling design used in PIRLS, students were assigned 
sampling weights. In general, the sampling weight assigned to a student was the 
inverse of the probability that the student would be selected for the sample. When 
responses were weighted, each contributed to the results for the total number of 
students represented by the individual student assessed. Weighting also adjusted for 
school and student nonresponse. The internationally defined weighting specifications 
for PIRLS required that each assessed student’s sampling weight should be the 
product of 


e@ the inverse of the school’s probability of selection; 

@ an adjustment for school-level nonresponse; 

e@ the inverse of the classroom’s probability of selection; and 
@ an adjustment for student-level nonresponse. 


Sampling weights should be used in all PIRLS analyses. A detailed description of this 
process is provided in PIRLS 2016 User Guide for the International Database (Foy 2018). 
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U.S. PIRLS and ePIRLS 2016 Data 


The PIRLS 2016 international databases contain student achievement data as well as 
student, teacher, and school background data for 61 education systems for PIRLS 
2016. These databases provide comparable data across education systems on detailed 
measures of student achievement in reading for PIRLS participants; information on 
educational practices and student outcomes; links between student achievement and 
background information from students, teachers, school principals, and curriculum 
experts; and achievement scales on a metric that is common to all cycles of PIRLS, 


allowing for the analysis of trends. 


U.S. International, and National Data Files 


The PIRLS 2016 national data for the United States exist in the following three 


forms: 


e US. international data files, which are part of the PIRLS international 
database and are directly comparable to those of other nations. These files 


allow for comparisons of the United States with any of the other education 


systems participating in PIRLS in virtually all respects. These files are 
available from the TIMSS & PIRLS International Study Center as SAS 


Note that these data files do not include the U.S.-specific adaptations made 
to a few questions in the questionnaires or the additional questions added to 
the school and student questionnaires, such as the question on 
race/ethnicity added to the student questionnaire. Furthermore, there are 
also restricted variables designated by IEA that are not available in the 
international public-use data files. Exhibit 5.1 below outlines the variables 


removed from the international public-use files. 
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Exhibit 5-1. Variables removed from public-use version of PIRLS 2016 international database 


File names Description 

ITBIRTHY Students’ year of birth from the tracking forms 
ITBIRTHM Students’ month of birth from the tracking forms 
ASBG02A Students’ year of birth from the student questionnaire 
ASBG02B Students’ month of birth from the student questionnaire 
ITDATE PIRLS 2016 test administration date 

ACBG01 Total school enrollment 

ACBG02 School enrollment in the target grade 

ITMODE_C Administration mode for school context questionnaires 
ITMODE_H Administration mode for home context questionnaires 
ITMODE_T Administration mode for teacher context questionnaires 


SOURCE: Foy, P. (Ed.) (2018). PIRLS 2016 User Guide for the International Database. Chestnut Hill, MA: TIMSS & PIRLS International 
Study Center, Boston College, exhibit 4-2, p. 52. 


e U.S. national public-use data files, which include the U.S.-specific 
adaptations that are not part of the U.S. international data files, some of 
which also include restricted variables designated by the IEA. These 
adaptations affect only a few variables and include the clarification or 
addition of response options on some international items and the addition 
of questions to each of the questionnaires. These additional questions are 
described in section 5.9.2 of this report. The U.S.-specific PIRLS data files 
are available separately from NCES by download from 
https://nces.ed.gov/surveys/pirls/datafiles.asp. These national files are in 
ASCII format and are named as indicated in exhibit 5-2. SAS and SPSS 
codes for reading these data files can also be downloaded from the NCES 


website.* 


Because the U.S. national files differ little from their international counterparts, the 
bulk of the variables describing U.S. students, teachers, and schools are as described 
in the PIRLS 2016 User Guide for the International Database (Foy 2018). These are the 
most comprehensive and detailed references for the PIRLS 2016 data and should be 


considered the primary references. 


e U.S. national restricted-use data files, which can only be obtained by 
completing a restricted-use license agreement with NCES. The restricted- 
use data files are provided only on CD-ROM. These datasets contain the 
supplemental link files that link PIRLS school ID numbers to the school ID 
numbers as they appear in the publicly available CCD or PSS. In addition, 


race/ethnicity is provided with all available categories, and free or 


8 The public-use dataset and restricted-use data CD-ROM contain a quick guide that explains how to 
use ASCII files with syntax files to create SPSS or SAS datasets. 
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reduced-price lunch is provided as a continuous variable. Because these data 
can reveal the identities of participating schools, the restricted-use data files 
are only made available to those who obtain an NCES restricted-use data 


license. Directions on how to obtain the license can be found at 


https://nces.ed.gov/pubsearch/licenses.asp. 


The discussion that follows is designed to provide the following: 


@ asummary overview of sections of the PIRLS 2016 User Guide for the 
International Database (Foy 2018) as these relate to using the U.S. data files; 


@ documentation of U.S. national items that differ from their international 


versions in the U.S. and state files; and 


e adescription of U.S. national items that are unique to the U.S. national and 


state files. 


5.2 PIRLS and ePIRLS Data Files 


The following five basic types of data files are available for each education system in 
both the PIRLS and ePIRLS international datasets: 


e achievement files containing item response data and scale scores for the 
PIRLS and ePIRLS assessment; 


e@ background files with information from students, from their reading 
teachers in PIRLS and ePIRLS, and from the principals of their schools; 


e student-teacher linkage files that contain the information needed to link data 


on students to that of their teachers; 


@ constructed-response scoring reliability files providing data on the reliability 
of scoring for this type of item; and 


@ curriculum data files that contain the responses of countries or participating 


education systems to the curriculum questionnaires. 


The naming convention of these files can be seen in Exhibit 5-2. It should be noted 
that the restricted-use versions of PIRLS and ePIRLS only contain 2 files each, as 
only the background files from students and curriculum data files have restricted-use 
variables. The following discussion focuses on the first three categories of files 


because these are the ones most likely to be used in data analyses by most users. For 
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the remaining two categories of files, the reader is referred to the PIRLS 2016 User 
Guide for the International Database (Foy 2018). 


Exhibit 5-2. U.S. national PIRLS and ePIRLS data file names 


Contents File name 


PIRLS 2016: Public-Use 


Student Background P4_STUDENT16.DAT 
Student Achievement P4_ACHIEVE16.DAT 
Teacher Background P4_TEACHER16.DAT 
Student-Teacher Linkage P4_STD_TCH_LINK16.DAT 
School Background P4_SCHOOL16.DAT 


ePIRLS 2016: Public-Use 


Student Background eP4_STUDENT16.DAT 
Student Achievement eP4_ACHIEVE16.DAT 
Teacher Background eP4_TEACHER16.DAT 
Student-Teacher Linkage eP4_STD_TCH_LINK16.DAT 
School Background eP4_SCHOOL16.DAT 


PIRLS 2016: Restricted-Use 
Restricted-use School Variables P4_RESTRICTED_USE16_SCH.DAT 
Restricted-use Student Variables P4_RESTRICTED_USE16_STD.DAT 


ePIRLS 2016: Restricted-Use 
Restricted-use School Variables eP4_RESTRICTED_USE16_SCH.DAT 
Restricted-use Student Variables eP4_ RESTRICTED _USE16_STD.DAT 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Progress in International Reading Literacy Study (PIRLS), 
2016. 


5.3. PIRLS and ePIRLS Achievement Data Files and Variable 
Names 


The data files containing the IRT-scaled achievement scores for overall reading and 
the several reading domains are identified by the first three characters in the file 


name. A set of five plausible values characterizes each of these achievement scores. 


For analytic convenience, these same achievement scores are also provided as an 
addition to the student background data files. 


The achievement score variable names are based on an eight-character string defined 
below. In exhibit 5-3, these conventions are illustrated by reference to the first 
plausible value for each of the total, content domain, and cognitive domain 
achievement scales in PIRLS and ePIRLS. 
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First character of the variable name (ASRREAO1): 
e A -— fourth-grade score 


Second character of the variable name (ASRREAO1): 


e S-—indicates that this is a student-level variable 
Third character of the variable name (ASRREAO1): 
e R-—PIRIS score 
e E-ePIRLIS score 
Fourth through sixth characters of the variable name (ASRREAO1): 


e A three-character code describing the achievement scale, as shown in 
exhibit 5-3 


Seventh and eighth characters of the variable name (ASRREAO1): 


e A two-digit number that indicates the plausible value — “01,” “02,” “03,” 
“04,” or 65,’ 


For example, ASRREAO1—ASRREAO05 represent the plausible values for PIRLS 
overall reading, while ASEREA01—ASEREA05 represent the plausible values for 
ePIRLS overall reading. Exhibit 5-3 provides the three-digit codes identify reading 


| | 


scales and subscales used in PIRLS and ePIRLS. 


Exhibit 5-3. Three-digit codes identifying achievement scales in PIRLS and ePIRLS 


Examples of variable names 


Score 

Achievement Scale identifier PIRLS ePIRLS 
Reading overall score REA ASRREAO1 ASEREAO1 
Purposes for reading 

Literacy experience LIT ASRLITO01 — 

Acquire and use information INF ASRINFO1 —_— 
Processes of comprehension 

Retrieving and straightforward inferencing RSI ASRRSIO1 ASERSIO1 

Interpreting, integrating, evaluating IIE ASRIIE01 ASEIIE01 


— Not available. 


SOURCE: Foy, P. (Ed.). (2018). P/IRLS 2016 User Guide for the International Database. Chestnut Hill, MA: TIMSS & PIRLS International 


Study Center, Boston College. 
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5.3.2 


5.4 


5.4.1 


5.4.2 
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PIRLS and ePIRLS Benchmark Achievement Variables 


The PIRLS achievement files also contain a set of variables indicating which 
international benchmark the students reached. For PIRLS and ePIRLS, the overall 
reading scales have five plausible values for each of the four benchmark levels 
defined (advanced, high, intermediate, low). The international benchmark variables 
follow the achievement score variable naming convention but substitute the letters 
“TBM” in the fourth through sixth positions of the variable name. Thus, ASRIBM01, 
ASRIBM02, ASRIBM03, ASRIBM04, and ASRIBMO05 are the five benchmark 
variables describing the fourth-grade overall reading score. It follows that 
ASEIBM01, ASEIBM02, ASEIBM03, ASEIBM04, and ASEIBM05 ate the five 


benchmark variables describing the fourth-grade overall online reading score. 


ePIRLS Student Questionnaire Variables 


The ePIRLS fourth-grade achievement file also includes additional student 
background variables from the ePIRLS student questionnaire pertaining to students’ 


familiarity with computer usage. 


PIRLS and ePIRLS Background Questionnaire Data 
Files 


Student, teacher, and school files contain the responses to the questions contained in 
the respective background questionnaires administered in PIRLS 2016, along with a 
fourth file used to link the student and teacher background data appropriately when 


student and teacher files are merged. 


Student Background Data Files (ASG) 


The student background data files contain students’ responses to questions in the 
student questionnaire, along with students’ reading achievement scores (as plausible 
values). At the fourth grade, there was a single version of the student questionnaire 
for PIRLS and ePIRLS. The files also contain a number of identification variables, 
tracking variables, sampling and weighting variables, and derived variables that were 


used to produce some of the exhibits in the international reports. 


Teacher Background Data Files (ATG) 


Teachers of PIRLS and ePIRLS classes were administered a single teacher 
background questionnaire that had questions about their background and their 


teaching practices in the classes of sampled students. Each teacher was asked to 
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complete a questionnaire for each class taught that contained sampled students. The 
PIRLS and ePIRLS teacher background data files contain one record for each 
teacher and class combination. Responses to the single questionnaire administered to 
fourth-grade teachers of PIRLS and ePIRLS are found in the ATG files. 


In all of the teacher files, each teacher has a unique identification number 
(IDTEACH) and a link number (IDLINKA) specific to the class taught by the teacher 
and to which the information in the data record corresponds. The IDTEACH and 
IDLINK combination uniquely identifies, within an education system, a teacher 
teaching a specific class. For example, students linked to teachers identified by the 
same IDTEACH but different IDLINK are taught by the same teacher but in 
different classes. It is important to note that the teachers in question do not 
constitute a representative sample of teachers in an education system but rather are 
the teachers who taught a representative sample of students. To reflect this fact, for 
the most part, the teacher data should be analyzed only in conjunction with the 


student-teacher linkage data files and weighted with student sampling weights. 


School Background Data Files (ACG) 


The school background data files contain the responses of school principals to 
questions about school policy, resources, and environment asked in the school 
questionnaire. That file also contains a series of identification variables, link variables, 
and sampling variables. The school data files can be merged with the student data 
files by using the education system and school identification variables. Details of the 
merging procedure using the SPSS-linked IEA International Database (IDB) 
Analyzer or using SAS programs for PIRLS and ePIRLS are provided in the PIRLS 
2016 User Guide for the International Database (Foy 2018). 


Student-Teacher Linkage Data Files (AST) 


The PIRLS and ePIRLS 2016 student-teacher linkage data files contain information 
required to link the student and teacher data files. These files contain one entry per 
student-teacher linkage combination in the data. For instance, if three teachers are 
linked to a student, there are three entries in the file corresponding to that student. 
The sole purpose of the student-teacher linkage data files is to link teacher-level data 
with student-level data to perform appropriate student-level analyses where teacher 


characteristics are disaggregated over students. 
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5.4.5 Curriculum Questionnaire Data Files 


In addition to the background questionnaires, PIRLS also provides data on the 
curriculum of the participating education systems. For PIRLS and ePIRLS, there is a 
single file. 


5.5 Variable Naming Convention for Background Variables 


The background variable naming convention is based on a seven- or eight-character 
string defined below. These conventions are illustrated by reference to an item in the 
fourth-grade school questionnaire. This item asks principals to report the population 


size of the community in which the school is located. 
First character of the variable name (ACBGO5A): 
e A -— fourth-grade data 


Second character of the variable name (ACBGO5A): 


e C-school principal 
e ‘'T — teacher 
e S-—student 
Third character of the variable name (ACBGO5A): 
e N-a national or nationally adapted background variable 
e X-—a new national background variable 
e B —all international background variables in the questionnaire data files 
e D —all international derived variables 


Fourth character of the variable name (ACBGO05A): 


e G- general question 
e R-—dquestion related to reading 


Fifth through seventh characters of the variable name (ACBG05A): 


e Used to represent the sequential numbering of each question 
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Eighth character of the variable name (ACBG12AA): 


e Represents the item within a multipart question 


5.9.1. Summary Indices and Derived Variables 


The PIRLS and ePIRLS questionnaires often devote several questions to a single 
construct. In these cases, responses to the individual items were combined to create 
a derived variable. A PIRLS index is a special type of derived variable that assigns 
students to one of three levels—high, medium, or low—based on their responses to 
the component variables. These variables are described in detail in Supplement 3 of 
the PIRLS 2016 User Guide for the International Database (Foy 2018). 


5.6 Sampling and Weighting Variables 


Several sampling and weighting variables are included in the PIRLS and ePIRLS data 
files. They are listed and described below in conjunction with a discussion of how 
and when these weights are used. Because PIRLS uses a complex sampling design, 
sampling weights must be used to generate accurate population estimates. The 
sampling weights account for the sample design and any stratification or 
disproportional sampling of subgroups; they also include adjustments for 
nonresponse (see LaRoche, Joncas, and Foy 2017). 


As noted, the sample of students is not a simple random sample; therefore, students 
in the sample do not have an equal probability of selection. Sampling weights adjust 
for this unequal probability and, in so doing, provide for statistical estimates 
reflective of the student population from which the sample was drawn. Sampling 
weights also include adjustments for school and student nonresponse. All PIRLS and 
ePIRLS analyses require the application of sampling weights. Provisions for 
weighting data are a standard feature of virtually all software likely to be used in 


analyses. 


The sampling weights included in the PIRLS and ePIRLS 2016 data files are 
described in exhibit 5-4. (Note that teacher background data files do not have any 
sampling weight variables because the analysis of teacher variables requires the 
merging of the teacher data with the student data and the use of student sampling 
weight variables.) 
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Exhibit 5-4 PIRLS and ePIRLS sampling weight variables 


Variable names Descriptions 

TOTWGT Total student weight — sums to the national population 

SENWGT Student senate weight — sums to 500 in each education system 

HOUWGT Student house weight — sums to the student sample size in each education system 
TCHWGT Overall teacher weight 

SCHWGT School-level weight 


SOURCE: Foy, P., and Drucker, K. T. (Eds.) (2013). PIRLS 2011 User Guide for the International Database. Chestnut Hill, MA: TIMSS & 


PIRLS International Study Center, Boston College. 


The characteristics of PIRLS and ePIRLS sampling weight variables are as follows: 


e TOTWGT sums to the student population size in each education system 


999 


and is appropriate for “within-country”” analyses and cross-country analyses 


where the analyses are conducted country by country and compared. 


e SENWGT is a transformation of TOTWGT that results in a weighted 
student sample size of 500 in each education system. This weight may be 
appropriate for cross-country analyses that require each education system to 
have the same number of students rather than proportionately more 


students from larger education systems and fewer from smaller education 
systems, which is the case if TOTWGT is used. 


e HOUWGT, another transformation of TOTWGT, ensures that the 
weighted sample corresponds to the actual sample size in each education 
system. This can be important because TOTWGT inflates sample sizes to 
approximate the population size, and software systems that use the actual 
sample size to compute significance tests will give misleading results under 


these conditions. 


e TOTWGT, SENWGT, and HOUWGT are designed for use in student- 


level analyses from all student-level files. 


e TCHWGT is specifically designed for analyses that link teacher background 


data to student data and is also used for analyses using all teachers. 


e SCHWGT is designed for use in school-level analyses where the schools are 


the units of analysis. 


° Although traditional terms with “country” are used throughout this discussion, they apply as well to 
all participating states and other education systems. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


5. U.S. PIRLS AND EPIRLS 2016 DATA | 83 | 


5.7 Structure and Design Variables in PIRLS and ePIRLS 
2016 Data Files 


The PIRLS and ePIRLS 2016 data files also contain unique numerical identification 
variables for each respondent along with sample design information. 
5.7.1 Identification Variables 


In all PIRLS and ePIRLS data files, identification variables are included to label 
countries, students, teachers, or schools. These variables also are used to link cases 
between the different data file types. The identification variables have the prefix 
“TD” and are described below. 


e IDCNTRY: a five-digit country identification code based on the ISO 3166 


classification. 
e IDPOP: identifies the target grade; “1” for the fourth grade. 
e IDGRADE: identifies the target grade of the participating students, “4.” 


e IDSCHOOL: a four-digit identification code that uniquely identifies the 


participating schools within each country but is not unique across countties. 


e IDCLASS: a six-digit identification code that uniquely identifies the sampled 


classrooms within a country. 


e IDSTUD: an eight-digit identification code that uniquely identifies each 


sampled student in a country. 


e IDBOOK: identifies the specific assessment booklet that was administered 
to each student. 


e IDTEACH: a six-digit identification code that uniquely identifies a teacher 


within a school. 


e IDLINK: uniquely identifies the class for which a teacher answered a 


questionnaire. 


5.7.2 Tracking Variables 


Information about students, teachers, and schools provided by the survey tracking 
forms described earlier is stored in the tracking variables. These variables have the 
prefix “IT.” ITBIRTHM, ITBIRTHY, and ITDATE are not included in the public- 
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9.8 


9.9 


use international files as mentioned in section 5.1 and are available upon request in 
the restricted-use files. TTLANG is included in the student achievement and student 
background data files. 


e ITSEX: gender of each student as stated in the Student Tracking Forms. 


e ITBIRTHM and ITBIRTHY: month and year of birth of each student as 
stated in the Student Tracking Forms. 


e ITDATE: date of testing for each student. 


e ITLANG: language of testing for each student. 


PIRLS and ePIRLS 2016 Codebook Files 


All information related to the structure of the PIRLS and ePIRLS 2016 data files, as 
well as the source, format, descriptive labels, and response option codes for all 
variables, is contained in codebook files. Each data file type in the database is 
accompanied by a codebook file, with the exception of the curriculum data files. 
These files are available from the PIRLS International Study Center website at 


U.S. National Instrumentation 


As noted earlier, the U.S. national instrumentation differs from the international 


instrumentation in five ways: 


e Minor language/expression adaptations were made to some of the 


instructions. 


e Minor language adaptations were made to the wording of some assessment 


items. 


e Fora few questionnaire items, response alternatives were changed but in a 


way that allowed a crosswalk to the international response alternatives. 


e Several U.S.-specific questions without international counterparts were 


added to the student and school questionnaires. 


Otherwise, the U.S. instrumentation is exactly the same as the international 


instrumentation. This will become apparent in comparisons between U.S. and 
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international questionnaires. For standard international versions of PIRLS 2016 
questionnaires, please see Supplement 1 of the PIRLS 2016 User Guide for the 
International Database (Foy 2018) PIRLS provides parallel supplements to the 


international user guides. 


The PIRLS supplements to the PIRLS 2016 User Guide for the International Database ate 


as follows: 


e Supplement 1, International Version of the PIRLS 2016 Background 
Questionnaires and Curriculum Questionnaire, can be downloaded from the 
TIMSS & PIRLS International Study Center at 


https: //timssandpirls.bc.edu/pirls2016/international- 


database/downloads/P16 UG Supplement1.pdf. 


e@ Supplement 2, the National Adaptations of International Background 
Questionnaires of the PIRLS 2016 User Guide for the International Database can 
be downloaded from the TIMSS & PIRLS International Study Center at 
https://timssandpirls.bc.edu/pirls2016/international- 


database/downloads/P16 UG Supplement2.pdf. 


e Supplement 3, the Variables Dertved from the Student, Home, Teacher, and 
School Questionnaire Data of the PIRLS 2016 User Guide for the International 
Database can be downloaded from the TIMSS & PIRLS International Study Center 
at https://pirls.bc.edu/pirls2016 /international- 
database/downloads/P16_ UG Supplement3.pdf. 


e The USS. versions of the PIRLS questionnaires may be downloaded from 
https://nces.ed.gov/surveys/pirls/questionnaire.asp. The variable names in 
the U.S. files are identical to those in the international files with the 


exception of the U.S.-specific variables added, and are included in this 
technical report. 


5.9.1 Background Questionnaire Items With U.S. Adaptations to Response 
Alternatives 


As the description of U.S. national adaptations in appendix D makes clear, there 
were a number of relatively minor changes to the wording of the international item 
stems and response alternatives in the questionnaires. Most of these adaptations do 
not require comment, as they are identical in format between the international and 
USS. versions of the questionnaires (for example, they contain simple wording 
changes). In some cases, however, the adaptations resulted in item response formats 


not immediately comparable between the international and national versions of the 
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questionnaires. As indicated in appendix D, there are instances in which the 
international and U.S. versions of variables have different sets of response codes; 


“highest level of formal education” in the teacher questionnaire is one example. 


This means that, for these items, the data will not be identical in international and 
USS. versions of the data files. Using the same example, “highest level of formal 
education” will have six response categories in the U.S. international file and seven 
categories in the U.S. national file. However, as indicated in appendix D, crosswalks 
between international and U.S. versions of these questions allow for the conversion 


of the U.S. response codes to the international format. 


5.9.2 U.S.-Specific Variables 


U.S.-specific items were added to the student, teacher, and school questionnaires. 


Four questions were added to the student questionnaires: 
1. a two-part question designed to collect the student’s race/ethnicity; 
2. a question that asked for language other than English spoken at home; 


3. a three-part question asking about students’ place of birth and parents’ place 
of birth; and 


4. a question about additional activities outside of school. 
Four questions were added to the school questionnaire: 
1. the percentage of students in the school eligible for free or reduced-price lunch; 
2. the percentage of students in the school who are English language learners; 
3. the average income level of the population in the school area; and 
4. a specification of the type of school. 


5.9.2.1 Race/ethnicity (Student Questionnaire) 


Students’ race/ethnicity was obtained through student responses to a two-part 
question in the student questionnaire. Students were asked first whether they were 
Hispanic or Latino and then whether they were members of the following five 
racial groups: (1) American Indian or Alaska Native; (2) Asian; (3) Black or African 
American; (4) Native Hawaiian or other Pacific Islander; or (5) White. Multiple 
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'° A composite variable 


responses to the second of these questions were allowed. 
with six categories was constructed in which results are shown separately for 

(1) Hispanics of any race; (2) Blacks; (3) Whites; (4) Asians; and (5) multiracial. The 
sixth category was labeled as “Other” and consisted of the small numbers of 
students indicating that they were American Indian or Alaska Native and Native 
Hawaiian or other Pacific Islander.'' The restricted-use student background file 
contains an uncollapsed version of this variable that separates out the “Other” 
category into the American Indian or Alaska Native and Native Hawaiian or other 


Pacific Islander response options. 


5.9.2.2 Language other than English spoken at home (Student Questionnaire) 


This item extended the international question about how often students spoke 
English at home by asking those who indicated that they did not always speak 
English if they spoke Spanish or another language at home. If another language 
other than Spanish was selected, they were asked to specify what language at home 
was spoken. Two variables were created from this national item. The first variable 
(ASXBG03B) is a categorical variable showing the students who selected (1) Spanish 
or (2) Other." The second variable (ASXBG03Ba) is a string variable with all the text 
responses entered by the students. For the text responses, responses that were 
considered invalid, such as numbers, “pig Latin,” and “gibberish,” were removed and 


set to missing. This was the case for 65 text responses. Misspellings were left as is.'* 


5.9.2.3 Additional outside activities (Student Questionnaire) 


The measure of outside of school activities was collected using a prompt with four 
yes/no questions. The prompt states, “The following questions ask about activities 


you do outside of school.” The yes/no questions were as follows: 


e Do you play on a sports team outside of school? 
e@ Do you play a musical instrument outside of school? 


e Are you studying something in a class outside of school? 


10 Race/ethnicity is provided with all categories in the restricted-use dataset. 

1! Race/ethnicity is provided as a composite variable in the public-use dataset. 

'2 The data suggest that some students may not have fully comprehended the instructions in the questions. 
Students who responded to the international item that they “always” speak English at home (ASBG03) were 
supposed to skip the item asking about other languages at home (ASXBG03B). However, about 15.3 percent 
of students who took PIRLS (677 cases) responded that they “always” speak English at home and also 
identified that they speak another language at home. In some cases, students specified “ENGLISH” as their 
other language. Therefore, results from the two variables (ASXBG03B and ASXBG03Ba) should be 
interpreted with caution. 
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e Do you belong to a club outside of school (like Girl Scouts, Cub Scouts, 
4-H, or Boys and Girls Club)? 


5.9.2.4 Birth country of mother/father/student (Student Questionnaire) 


Students were asked about the location of their birth and about the location of their 
parents’ or legal guardians’ birth using three yes/no questions. These questions were 
as follows: 


A. Was your mother (or stepmother or female legal guardian) born in the 
United States? (“United States” includes the 50 states, its territories, the 
District of Columbia, and U.S. military bases abroad.) 


B. Was your father (or stepfather or male legal guardian) born in the United 
States? 


C. Were you born in the United States? 


5.9.2.5 Poverty level in public schools (percentage of students eligible for free or 
reduced-price lunch) (School Questionnaire) 


The measure of poverty level in public schools was obtained from principals’ 
responses to the school questionnaire. The question asked the principal to report, as 
of approximately October 2015, the percentage of students at the school eligible to 
receive free or reduced-price lunch through the National School Lunch Program. 
Responses were grouped into five categories: (1) less than 10 percent, (2) 10 to 24.9 
percent, (3) 25 to 49.9 percent, (4) 50 to 74.9 percent, and (5) 75 percent or more.” 
Missing data on this variable were replaced with measures taken from the Common 
Core of Data (CCD). The effect of this replacement on the confidentiality of the 
data was examined as part of the confidentiality analyses described in section 4.6.2. 
The restricted-use school file contains a scale version of this variable, rather than 


categorical. 


5.9.2.6  Limited-English proficient/English language learners (School Questionnaire) 


Principals were asked to report the percentage of students with limited English 
proficiency or English language learners. They were provided with the following 
eight response categories: 0 percent; 1-5 percent; 6—10 percent; 11—25 percent; 
26-50 percent; 51—75 percent; 76—90 percent; and over 90 percent. 


13 Pree or reduced-price lunch is provided as a composite variable in the public-use dataset and as a 
continuous variable in the restricted-use dataset. 
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5.9.2.7 Type of school (School Questionnaire) 


Principals were asked to identify their schools using one of the following 10 response 
categories: (1) regular public school; (2) regular public school with magnet 
program;(3) magnet school or school with special program; (4) special education; 

(5) alternative curriculum; (6) vocational; (7) charter school; (8) independent private 
school; (9) religiously affiliated private school; or (10) other school. 


5.9.2.8 Missing data 


Data derived from the student, school, and teacher questionnaires and from the 
student assessments contain missing data in varying amounts. Four sources of 


missing data are identified: 


1. Not administered. The respondent was not administered the actual item. He 


or she had no chance to read and answer the question. 


2. Omitted or invalid. The respondent had a chance to answer the question but 
did not do so. This code also was used for responses that were not 


interpretable. 


3. Logically not applicable. The respondent answered a preceding filter question 
in a way that made the following dependent questions not applicable to him 


ot het. 


4. Not reached (only used in the achievement files). This code indicates those 
items not reached by the students due to a lack of time. 


SAS and SPSS control code for all the data files include the code for 


handling/converting missing data. 


5.9.2.9 Imputation 


No imputation for missing values was done. However, missing data on the measure 
of school poverty (proportion of students eligible for free or reduced-price lunch) 


reported by schools was replaced as described above. 


5.10 Merging PIRLS 2016 Data Files 


In preparing PIRLS and ePIRLS 2016 data for analysis, it may be necessary to merge 
two (or more) of the data files named in exhibit 5-4. Not every analysis will require 


merging of files however. For example, analyses looking at the relationship between 
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student background and achievement can be done using the student background file 
alone. However, analyses that wish to examine the relationships between school, 
teacher or student characteristics and student achievement will require that files be 
merged. Standard merging procedures as implemented in SPSS, SAS, or Stata can be 
applied. Examples are provided below along with illustrative SAS and SPSS code. 
(These various merges are facilitated for SPSS users who choose to work with the 
IEA International Database Analyzer [IEA IDB Analyzer] described below.) 


The merging procedures illustrated below follow the same pattern as previous PIRLS 
studies (see Foy and Drucker 2013) and are illustrated with PIRLS fourth-grade data. 


5.10.1 Merging Student and School Data 


If the intent is to disaggregate school data across students, the school-level data are 
merged to the student file using IDSCHOOL. The disaggregated data can be 
analyzed at the student level using the student-level weight TOTWGT. Exhibits 5-5 
and 5-6 provide examples of how to merge the student and school data using SAS 
and SPSS. Additional examples are provided in chapters 2 and 3 in the PIRLS 2016 
User Sia for the International Database (Foy 2018; available at 


Exhibit 5-5. Illustrative SAS code for merging U.S. PIRLS student and school data 
libname aR4 "C:\PIRLS2016\Data"; 


data SCHOOL; 
set aR4.P4_SCHOOL16; 


proc sort data= SCHOOL; 
by IDSCHOOL; 


data STUDENT; 
set aR4.P4_STUDENT16; 


proc sort data= STUDENT; 
by IDSCHOOL; 


data aR4.MERGE1; 

merge STUDENT SCHOOL; 
by IDSCHOOL; 

run; 


SOURCE: International Association for the Evaluation of Education Achievement (IEA), Progress in International Reading Literacy Study 


(PIRLS) 2016. 


This example creates a temporary SAS dataset (SCHOOL) using the permanent 
school dataset aR4.P4_SCHOOL16. It then sorts the school data by school ID 
(IDSCHOOL). A similar procedure is used for the student file (STUDENT), which 
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is also sorted by the school ID using the permanent student dataset 
aR4.P4_STUDENT16. The final dataset is a permanent dataset called 
aR4.MERGE1, containing the merged file from SCHOOL and STUDENT using 
IDSCHOOL as the merge variable. 


The SPSS example shown in exhibit 5-6 works in a similar way. SPSS uses a file 
containing the school variables (P4_SCHOOL16.SAV) and sorts the cases by 
IDSCHOOL. The same procedute is used for the student dataset, 
P4_STUDENT16.SAV. The “match files” command merges the two files, and the 
final, merged output file is saved as MERGE1.SAV. 


Exhibit 5-6. Illustrative SPSS code for merging U.S. PIRLS student and school data 


get file = "C:\PIRLS2016\Data\P4_SCHOOL16.SAV". 
sort cases by IDSCHOOL. 
save outfile = 'C:\PIRLS2016\Data\SCHOOL.SAV'. 


get file = "C:\PIRLS2016\Data\P4_STUDENT16.SAV". 
sort cases by IDSCHOOL. 
save outfile = 'C:\PIRLS2016\Data\STUDENT.SAV'. 


match files 

/ file= 'C:\PIRLS2016\Data\STUDENT.SAV' 

/ table= 'C:\PIRLS2016\Data\SCHOOL.SAV' 

/ by IDSCHOOL. 

save outfile = "C:\PIRLS2016\Data\MERGE1.SAV". 


SOURCE: International Association for the Evaluation of Education Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


5.10.2 Merging Student and Teacher Data 


In the United States, the student sample was based on intact classrooms. The 
teachers of the students selected in this way are not a sample of teachers and should 
be seen as the “teachers of the sampled students.” To maintain this linkage, merges 
of teacher and student data must use the student-teacher link file 
(P4_STD_TCH_LINK16.DAT), which also contains the appropriate teacher 
sampling weights. Exhibits 5-7 and 5-8 provide illustrative code for merging the 
student and the mathematics teacher files in SAS and SPSS. 
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Exhibit 5-7. Illustrative SAS code for merging U.S. PIRLS student and teacher data 
libname aR4 "C:\PIRLS2016\Data"; 


data TEACHER; 
set aR4.P4_TEACHER16; 


proc sort data = TEACHER; 
by IDTEACH IDLINkK; 


data STDTCH; 
set aR4.P4_STD_TCH_LINK16; 


proc sort data = STDTCH; 
by IDTEACH IDLINK; 


data TEACHMRG; 

merge TEACHER STDTCH; 
by IDTEACH IDLINK; 

if TCHWGT > 0; 


proc sort data = TEACHMRG; 
by IDSTUD; 


data STUDENT; 
set aR4.P4_STUDENT16; 


proc sort data = STUDENT; 
by IDSTUD; 


data aR4.MERGE2; 

merge STUDENT TEACHMRG; 
by IDSTUD; 

run; 


SOURCE: International Association for the Evaluation of Education Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


In the SAS example, the program creates a temporary SAS dataset (TEACHER) 
using the permanent teacher file, aR4.P4_TEACHER16. It then sorts the teacher 
data by the teacher ID DTEACH) and the link ID (IDLINKA). A similar procedure 
is used for the student-teacher link file SG TDTCH), using the permanent file 
(aR4.P4_STD_TCH_LINK16), which is also sorted by the teacher ID and the link 
ID. The weight variable for teachers (ICHWGT) is used as a selection variable 
because teachers have been selected. The result is a merged file called 
aR4.TEACHMRG with disaggregated teacher data. This file is merged with the 
student file STUDENT). The final dataset is a permanent dataset called 
aR4.MERGE2 that contains the merged file from TEACHMRG and STUDENT 
using IDSTUD as the merge variable. 
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The SPSS student-teacher merge in exhibit 5-8 uses a file containing the teacher 
variables (P4_TEACHER16.SAV) and sorts the cases by IDTEACH and IDLINK. 
The file is then saved as TEACHER. The same procedure is used for the student- 
teacher linkage dataset P4_STD_TCH_LINK16.SAV. The “match files” command 
merges the two files by the ID variables JOTEACH and IDLINK, and the merged 
output file is saved as TEACHMRG. To include the student data, the student file is 
selected (P4_STUDENT16.SAV), sorted by IDSTUD, and saved as STUDENT. 
This file is merged with TEACHMRG using IDSTUD to create the final file 
MERGE2.SAV containing both teacher and student variables. 


Exhibit 5-8. Illustrative SPSS code for merging U.S. PIRLS student and teacher data 


get file = "C:\PIRLS2016\Data\P4_TEACHER16.SAV". 
sort cases by IDTEACH IDLINK. 
save outfile = 'C:\PIRLS2016\Data\TEACHER.SAV'. 


get file = "C:\PIRLS2016\Data\P4_STD_TCH_LINK16.SAV". 
select if TCHWGT > 0. 

sort cases by IDTEACH IDLINK. 

save outfile = 'C:\PIRLS2016\Data\STDTCH.SAV'. 


match files 

/ file = 'C:\PIRLS2016\Data\STDTCH.SAV' 

/ table = 'C:\PIRLS2016\Data\TEACHER.SAV' 

/ by IDTEACH IDLINK. 

sort cases by IDSTUD. 

save outfile = 'C:\PIRLS2016\Data\TEACHMRG.SAV'. 


get file = "C:\PIRLS2016\Data\P4_ STUDENT16.SAV". 
sort cases by IDSTUD. 
save outfile = 'C:\PIRLS2016\Data\STUDENT.SAV'. 


match files 

/ file = 'C:\PIRLS2016\Data\TEACHMRG.SAV' 

/ table = 'C:\PIRLS2016\Data\STUDENT.SAV' 

/ by IDSTUD. 

save outfile = "C:\PIRLS2016\Data\MERGE2.SAV". 


SOURCE: International Association for the Evaluation of Education Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS), 2016. 


5.10.3 Merging Student, School, and Teacher Data 


In merging student, teacher, and school data together to form a single dataset, the 
procedures from sections 5.11.1 and 5.11.2 are combined. Exhibits 5-9 and 5-10 
show illustrative SAS and SPSS code designed to achieve this three-way merge. This 
example uses the same merging steps as with the previous school and teacher 
examples (MERGE1 and MERGE2), then merges the output files by the student ID, 
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IDSTUD, into a final file aR4. MERGEALL containing linked student, school, and 


teacher data at the student level. 


Exhibit 5-9. Illustrative SAS code for merging U.S. PIRLS school, teacher, and student data 
libname aR4 "C:\PIRLS2016\Data"; 


data SCHOOL; 
set aR4.P4_SCHOOL16; 


proc sort data = SCHOOL; 
by IDSCHOOL; 


data STUDENT; 
set aR4.P4_STUDENT16; 


proc sort data = STUDENT; 
by IDSCHOOL; 


data MERGE1; 
merge STUDENT SCHOOL; 
by IDSCHOOL; 


proc sort data = MERGE‘1; 
by IDSTUD; 


data TEACHER; 
set aR4.P4_TEACHER16; 


proc sort data = TEACHER; 
by IDTEACH IDLINkK; 


data STDTCH; 
set aR4.P4_STD_TCH_LINK16; 


proc sort data = STDTCH; 
by IDTEACH IDLINK; 


data MERGE2; 

merge STDTCH TEACHER; 
by IDTEACH IDLINK; 

if TCHWGT > 0; 


proc sort data = MERGE2; 
by IDSTUD; 


data aR4.MERGEALL; 
merge MERGE1 MERGE2; 
by IDSTUD; 

run; 


SOURCE: International Association for the Evaluation of Education Achievement (IEA), Progress in International Reading Llteracy Study 
(PIRLS) 2016. 
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In the SPSS example shown in exhibit 5-10, the student and school data are first sorted 
by IDSCHOOL and then merged. The procedure followed for combining student and 
teacher data in exhibit 5-9 is used again. Then the saved student-school and student- 
teacher files are merged by IDSTUD, and a final dataset MERGEALL.SAV is saved. 


Exhibit 5-10. Illustrative SPSS code for merging U.S. PIRLS school, teacher, and student data 


get file = "C:\PIRLS2016\Data\P4_SCHOOL16.SAV". 
sort cases by IDSCHOOL. 
save outfile = 'C:\PIRLS2016\Data\SCHOOL.SAV'. 


get file = "C:\PIRLS2016\Data\P4_STUDENT16.SAV". 
sort cases by IDSCHOOL. 
save outfile = 'C:\PIRLS2016\Data\STUDENT.SAV'. 


match files 

/ file = 'C:\PIRLS2016\Data\STUDENT.SAV' 

/ table = 'C:\PIRLS2016\Data\SCHOOL.SAV' 

/ by IDSCHOOL. 

save outfile = "C:\ PIRLS2016\Data\MERGE1.SAV" . 


get file = "C:\PIRLS2016\Data\P4_TEACHER16.SAV". 
sort cases by IDTEACH IDLINK . 
save outfile = 'C:\PIRLS2016\Data\TEACHER.SAV'. 


get file = "C:\PIRLS2016\Data\P4_STD_TCH_LINK16.SAV". 
select if TCHWGT >0. 

sort cases by IDTEACH IDLINK. 

save outfile = 'C:\ PIRLS2016\Data\STDTCH.SAV'. 


match files 

/ file = 'C:\PIRLS2016\Data\STDTCH.SAV' 

/ table = 'C:\PIRLS2016\Data\TEACHER.SAV' 

/ by IDTEACH IDLINK. 

save outfile = "C:\PIRLS2016\Data\MERGE2.SAV". 


Get file = "C:\PIRLS2016\Data\MERGE1.SAV". 
Sort cases by IDSTUD. 
save outfile = 'C:\PIRLS2016\Data\MERGE1.SAV'. 


Get file = "C:\PIRLS2016\Data\MERGE2.SAV". 
Sort cases by IDSTUD. 
save outfile = 'C:\PIRLS2016\Data\MERGE2". 


match files 

/ file = 'C:\PIRLS2016\Data\MERGE2.SAV' 

/ table = 'C:\PIRLS2016\Data\MERGE1.SAV' 

/ by IDSTUD. 

save outfile = "C:\PIRLS2016\Data\MERGEALL.SAV". 


SOURCE: International Association for the Evaluation of Education Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER’S GUIDE 


| 96 | 5. U.S. PIRLS AND EPIRLS 2016 DATA 


5.10.4 Merging PIRLS 2016 Data with Restricted-Use Data 


Users who have been granted a license to use the restricted-use PIRLS 2016 data will 
receive a restricted-use CD-ROM that contains an additional link file that provides a 
way to merge PIRLS data with school data from the CCD and PSS. The NCESSCH 
(the NCES unique public school identification code) from the PIRLS file is used to 
merge with NCESSCH from the CCD file. The PPIN (the private school’s unique 
identification number) from the PIRLS file is used to merge with the PPIN from the 
PSS file. Illustrattve SAS and SPSS code is provided in exhibits 5-11 and 5-12. 


The code in question provides for a link between the PIRLS school data and the CCD/PSS 
data by school. Further merging to other PIRLS files (student, school, teacher) can be 
conducted using the IDSCHOOL for merging, as has been shown in earlier examples. 


Exhibit 5-11. Illustrative SAS code for merging U.S. PIRLS school data with restricted-use (CCD 
and PSS) data 


libname aR4 "C:\PIRLS2016\Data"; 


data SCHOOL; 
set aR4.P4_RESTRICTED_USE16; 


proc sort data = SCHOOL; 
by NCESSCH; 


data CCD; 
set aR4.CCD; 


proc sort data = CCD; 
by NCESSCH; 


data MERGE‘1; 

merge CCD(IN=IN1) SCHOOL (IN=IN2); 
by NCESSCH; 

IF IN2; 

run; 


/* User can Merge in PSS data to the file containing CCD and PIRLS previously merged data */ 


Data SCHOOL2; 
Set MERGE1; 


proc sort data = SCHOOL2; 
by PPIN; 


data PSS; 
set aR4.PSS; 


proc sort data = PSS; 
by PPIN; 


data MERGE2; 

merge PSS(IN=IN1) SCHOOL2 (IN=IN2); 
by PPIN; 

IF IN2; 

run; 


SOURCE: International Association for the Evaluation of Education Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS) 2016. 
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SPSS code designed to provide this same link between the PIRLS school data and 
the CCD/PSS data by school is provided in exhibit 5-12. Further merging to other 
PIRLS files (student, school, teacher) can be conducted using the JIODSCHOOL for 
merging as has been shown in earlier examples. 


Exhibit 5-12. Illustrative SPSS code for merging U.S. PIRLS fourth-grade school data with 
restricted (CCD and PSS) data 


get file = "C:\PIRLS2016\Data\P4_RESTRICTED_USE16.SAV". 
sort cases by NCESSCH. 
save outfile = 'C:\PIRLS2016\Data\SCHOOL.SAV'. 


get file = "C:\PIRLS2016\Data\CCD.SAV". 
sort cases by NCESSCH. 
save outfile = 'C:\PIRLS2016\Data\CCD.SAV'. 


match files 

/ file = 'C:\PIRLS2016\Data\CCD.SAV' 

/ table = 'C:\PIRLS2016\Data\SCHOOL.SAV' 

/ by NCESSCH. 

save outfile = "C:\PIRLS2016\Data\MERGE1.SAV". 


* Merge PSS to the Combined PIRLS/CCD school-level file 


get file = "C:\PIRLS2016\Data\MERGE1.SAV". 
sort cases by PPIN. 
save outfile = 'C:\PIRLS2016\Data\SCHOOL2.SAV'. 


get file = "C:\PIRLS2016\Data\PSS.SAV". 
sort cases by PPIN. 
save outfile = 'C:\PIRLS2016\Data\PSS.SAV'. 


match files 

/ file = 'C:\PIRLS2016\Data\PSS.SAV' 

/ table = 'C:\PIRLS2016\Data\SCHOOL2.SAV' 

/ by PPIN. 

save outfile = "C:\PIRLS2016\Data\MERGE2.SAV" . 


SOURCE: International Association for the Evaluation of Education Achievement (IEA), Progress in International Reading Literacy Study 
(PIRLS), 2016. 


5.11 Some Notes on Analyzing the PIRLS 2016 Data 


The design of PIRLS raises three special considerations for the analysis of PIRLS 
data. First, the assessment design necessitates the use of five plausible values rather 
than a single score for each of the various measures of mathematics, science, and 
reading achievement. Second, since the sampling design is not a simple random 


sample in which each student had an equal probability of selection, sampling weights 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


| 98 | 5. U.S. PIRLS AND EPIRLS 2016 DATA 


must be applied to generate unbiased estimates of population parameters. Third, the 
complex sampling design also means that the calculation of the standard errors of the 


various statistics generated requires special procedures. 


5.11.1 Plausible Values 


As noted earlier, the assessment design was based on balanced incomplete block 
(BIB) spiraling of assessment items to increase content-area coverage without a 
concomitant increase in the assessment time demanded of students. Each student 
completed only a subset of the total pool of assessment items, with the resulting data 
containing missing values for other items in the pool but not in the subset 
administered to the student. The trade-off for increased coverage through BIB 
spiraling is increased measurement error in the scores available for each student. This 
is accommodated through the estimation of (five) plausible values for each student 
rather than a single (unreliable) point estimate. Plausible values are random draws 
from the estimated distribution of a student’s achievement. A detailed description of 
the PIRLS 2016 scaling can be found in Methods and Procedures in PIRLS 2016 (Mattin, 
Mullis, and Hooper 2017). 


What this means for those analyses of PIRLS data that include achievement 
measures is that the analyses need to be done five times and the results averaged. For 
example, if one was regressing mathematics achievement on a number of family and 
school attributes, it would be necessary to estimate this equation five times and then 
average each set of five parameter estimates. It would not be legitimate to take the 
mean of the five plausible values in the first instance and then regress this mean on a 


number of family and school attributes. 


5.11.2 Estimating Sampling Variance 


The complex sampling design used in PIRLS 2016 complicates the task of 
computing standard errors. Most standard analysis software systems such as SAS and 
SPSS provide estimates based on the assumption of a simple random sample. Given 
the PIRLS sampling design, such standard errors will underestimate the true standard 
errors. PIRLS adopt the jackknife repeated replication (JRR) technique because it is 
computationally straightforward and provides approximately unbiased estimates of 
the sampling errors of means, totals, and percentages. The variables necessary for 
these JRR procedures are included as part of the PIRLS 2016 data files: JKZONE, 
the sampling zone (stratum) of the student’s school; and JKREP, the sampling 
replicate (primary sampling unit) of the student’s school. 
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There ate, however, several options for estimating sampling errors that avoid the 
assumption of simple random sampling. The SPSS- and SAS-linked IDB Analyzer 
software was designed specifically by the IEA to analyze PIRLS international data 
files. This software is freely available from the IEA website at https://www.iea.nl & 


Special-use software is also available for estimating the standard errors of statistics 
generated from complex sampling designs. Among the packages available are AM, 
available from the American Institutes for Research at http://am.air.org/about2.asp & 
and SUDAAN, available from RTI International at https://www.tti.org/sudaan &. 
Some software packages provide for these capabilities as well. 


In addition, SAS macros suitable for this purpose are available as part of the PIRLS 
2016 User Guide for the International Database (Foy 2018). See also work by Stapleton 
(2006, 2008), which suggests procedures that can be used to generate appropriate 


standard errors for statistics generated by structural equation modeling techniques. 


5.11.3 IDB Analyzer and International Data Explorer (IDE) 


As described in section 5.11.2, the IDB Analyzer was developed by the IEA DPC as 
a plug-in for SAS and SPSS and can only be used in conjunction with SAS and SPSS. 


It is not a stand-alone analysis system. 


The IDB Analyzer enables users to combine SPSS data files and conduct analyses 
using SPSS without writing programming code. The IDB Analyzer generates SPSS 
syntax and SAS program that takes into account information from the sampling 
design in the computation of statistics and their standard errors. In addition, the 
generated SPSS syntax and SAS program makes appropriate use of plausible values 
for calculating estimates of achievement scores and their standard errors, combining 


both sampling variance and imputation variance. 


The IDB Analyzer consists of two modules—a merge module and an analysis 
module. The merge module is used to create analysis datasets by combining data files of 
different types and from different countries and selecting subsets of variables for 
analysis. The analysis module provides procedures for computing various statistics and 
their standard errors. All statistical procedures offered within the analysis module of 
the IDB Analyzer make appropriate use of sampling weights, and standard errors are 
computed using the JRR method. Percentages, means, regressions, and correlations 
may be specified with or without achievement scores. When achievement scores ate 
used, the analyses are performed five times—once for each plausible value—and the 
results ate aggregated to produce accurate estimates of achievement and standard 


errors that incorporate both sampling and imputation errors. 
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The use of the IDB Analyzer is described in detail with worked examples in chapter 
2 of the PIRLS 2016 User Guide for the International Database (Foy 2018). Readers 


intending to use this user-friendly software are urged to read this user guide in detail. 


In addition to IDB Analyzer for basic analysis and exploration of PIRLS data, NCES 
has developed a relatively simple, interactive online data-analysis tool: the 
International Data Explorer (IDE) can be found at 
https://nces.ed.gov/surveys/international/ide/, The IDE allows users to analyze all 
the international variables for all participating education systems and the U.S.- 
specific variables; however, it does not include U.S. restricted-use data. The IDE 
does not require SPSS or SAS for analyzing the data. It provides users with the 
capabilities to create statistical tables and charts of PIRLS data across countries and 
yeats on the website. This tool allows users to point and click in a self-contained 
module, unlike the IDB Analyzer software that must be used in conjunction with 
SPSS. Also, unlike the IDB Analyzer, the IDE does not provide access to data files 
for merging, transforming, or otherwise manipulating data. This tool reports 
averages for subject by selected variables and exports reports in HTML, Excel, 
Word, or PDF. 


5.11.4 SAS Programs and Macros 


The PIRLS 2016 User Guide for the International Database also provides assistance for 
those investigators who wish to conduct their analyses using SAS. The user guide 
includes a number of SAS programs needed to process the SAS data files, compute 
sutvey results, and carry out example analyses. These are described in detail with 
worked examples in chapter 3 of the user guide. Readers intending to use SAS for 
their analyses are urged to read this chapter in detail. 


The following SAS programs and macros are available: 


e P16_CONVERT-SAS and eP16_CONVERT are used to convert SAS 
export files into SAS data files for PIRLS and ePIRLS, respectively. 


e@ ASASCRR4.SAS and ASACRE1.SAS are used to convert the response 
codes on the achievement items to their corresponding score levels for 
PIRLS and ePIRLS, respectively. 


e P16_PL16_COMBINE.SAS combines merged PIRLS 2016 files to the 
PIRLS Literacy 2016 data files. 
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5.11.5 Special Considerations in Using the Teacher Data 


The teachers in the PIRLS 2016 international databases are the teachers of nationally 
representative samples of students and are not representative samples of teachers in 
the participating countries. As a result, analyses with teacher data should be made 
with students as the units of analysis and reported in terms of students who ate 


taught by teachers with a particular attribute. 


When analyzing teacher data, it is first necessary to link the students to their 
respective teachers. The student-teacher linkage data files (AST) were created for this 
purpose. Because student achievement scores (plausible values), jackknife replication 
information, and teacher weighting variables are found in the student-teacher linkage 
data files, it is only necessary to merge the teacher background data files with the 
student-teacher linkage data files. For analyses linking teacher variables to student 
background variables, it is also necessary to merge the student background data files 
with the teacher background data files after they have been combined with the 
student-teacher linkage data files. 


In general, to perform analyses using the teacher background data files, follow the 


steps below. 


1. Identify the variables of interest in the teacher background data files and note 


any specific national adaptations to the variables. 


2. Retrieve the relevant variables from the teacher background data files, 
including analysis variables, classification variables, identification variables 
(IDCNTRY, IDTEACH, and IDLINWA), and any other variables used in the 


selection of cases. 


3. Retrieve the relevant variables from the student-teacher linkage data files, 
including plausible values of achievement, classification variables, 
identification variables IDCNTRY, IDSTUD, IDTEACH, and IDLINk), 
sampling (KZONE and JKREP) and weighting (TCHWGT) variables, and 


any other variables used in the selection of cases. 


4. Merge the teacher background data files with the student-teacher linkage data 
files using the variables IDCNTRY, IDTEACH, and IDLINK. 


5. If student background variables also are needed, merge the student 
background data files with the merged student-teacher data files from the 
previous step using the variables IDCNTRY and IDSTUD. 
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5.11.6 Special Considerations in Using the School Data 


In general, to perform analyses using the school background data files, follow the 


steps below. 


1, 


Identify the variables of interest in the school and student background data 


files and note any specific national adaptations to the variables. 


Retrieve the relevant variables from the school background data files, 
including analysis variables, classification variables, identification variables 
(IDCNTRY and IDSCHOOL), and any other variables used in the selection 


of cases. 


Retrieve the relevant variables from the student background data files, 
including plausible values of achievement, classification variables, 
identification variables IDCNTRY and IDSCHOOL), sampling JKZONE 
and JKREP) and weighting (TOTWGT) variables, and any other variables 


used in the selection of cases. 


Merge the school background data files with the student background data 
files using the variables IDCNTRY and IDSCHOOL. 
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Exhibit A-1. PIRLS 2016 State Contacting Letter 


[Date] 

[Title] [Name First] [Name Last] 
[Title/Department] 

[State] 

[Address 1] 

[Address 2] 

[City], [State] [Zip code] 


Dear [Title] [Name Last]: 


The United States is participating in an important international study in 2016: the Progress in International 
Reading Literacy Study (PIRLS). Since 2001, PIRLS has measured trends in reading achievement at grade 4 in 
countries around the world, including the United States. Results from PIRLS are used by researchers and 
policymakers to chart national progress against international standards and other countries around the world, 
informing national discussions about international competitiveness. 


Some schools in your state have been randomly selected to participate in PIRLS in spring 2016. For the first 
time PIRLS will also include an innovative assessment of online reading called ePIRLS. Some classrooms 
selected to participate in PIRLS may also be asked to take part in ePIRLS. I am writing to ask your agency to 
support the participation of those selected schools. 


PIRLS is described in more detail in the enclosed materials. In the United States, PIRLS is conducted by the 
National Center for Education Statistics (NCES), part of the U.S. Department of Education, and the data are 
being collected by RTI International. The U.S. Office of Management and Budget has approved the data 
collection under OMB #1850-0645. For information on the confidentiality of the data collected, please see 
the enclosed FAQ. While participation in this study is voluntary, your support of school participation in your 
state is invaluable so that the United States has a representative sample of schools across the country. 


Within the next few weeks, a representative of RTI International will contact sampled school districts and 
schools to discuss participating in the assessment. In the meantime, if you have questions about the study, 
please do not hesitate to call Dr. Patricia Green at (XXX) XXX-XXXX or send an email to PIRLS@rti.org. 
You may also get more information about this study by contacting Dr. Sheila Thompson at NCES at (202) 
502-7425 or sheila.thompson@ed.goy, or by visiting the PIRLS website at 
https://nces.ed.gov/sutveys/PIRLS/index.asp. 


Thank you for your time and support of this important international study. 


Sincerely, 
Sue Betka Pegey Carr 
Acting Director, Institute of Education Sciences Acting Commissioner, NCES 


cc: [State assessment director] 
Enclosures 


NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002; 20 U.S.C. § 9543). By law, the data provided 
by schools, staff, and students may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as 
required by law (20 U.S.C. § 9573). 
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Exhibit A-2. PIRLS 2016 District Contacting Letter 


[Date] 

[Title] [Name First] [Name Last], [Title/Department] 
[School District] 

[Address 1] 

[City], [State] [Zip code] 


Dear [Title] [Name Last]: 


The United States is participating in an important international study in 2016: the Progress in International 
Reading Literacy Study (PIRLS). Since 2001, PIRLS has measured trends in reading achievement at grade 4 in 
countries around the world, including the United States. Results from PIRLS are used by researchers and 
policymakers to chart national progress against international standards and other countries around the world, 
informing national discussions about international competitiveness. 


One or mote schools in your district have been randomly selected to participate in PIRLS in spring 2016. For 
the first time PIRLS will also include an innovative assessment of online reading called ePIRLS. Some 
classrooms selected to participate in PIRLS may also be asked to take part in ePIRLS. I am writing to ask 
your agency to support the participation of those selected schools. 


Participating schools will receive $200, and each school’s PIRLS school coordinator (the school staff person 
designated to work with PIRLS staff) will receive $100 as a thank you for his or her time and effort. The 
school coordinator may also receive an additional monetary token of appreciation of $50 for assistance with 
ePIRLS. A school administrator and selected teachers will each be asked to complete a questionnaire. 
Teachers will receive $20 as a thank you for completing the questionnaire. Each student who participates will 
receive a small gift as a token of appreciation. 


PIRLS is described in more detail in the enclosed materials. In the United States, PIRLS is conducted by the 
National Center for Education Statistics (NCES), part of the U.S. Department of Education, and the data are 
being collected by RTI International. The U.S. Office of Management and Budget has approved the data 
collection under OMB #1850-0645. For information on the confidentiality of the data collected, please see 
the enclosed FAQ. While participation in this study is voluntary, your support of school participation in your 
district is invaluable so that the United States has a representative sample of schools across the country. 


Within the next few days, a representative of RTI International will contact the following school or schools in 
your district that have been selected for the assessment: [LIST SAMPLED SCHOOLS HERE...]. 


If you have any questions about the study, please do not hesitate to call Dr. Patricia Green at (XXX) XXX- 
XXXX or send an email to PIRLS@rti.ore. You may also get more information about this study by 
contacting Dr. Sheila Thompson at NCES at (202) 502-7425 or sheila.thompson@ed.gov, or by visiting the 


PIRLS website at https://nces.ed.gov/surveys/PIRLS /index.asp. 


Thank you for your time and support of this important international study. 


Sue Betka Pegey Carr 
Acting Director, Institute of Education Sciences Acting Commissioner, NCES 
Enclosures 


NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002; 20 U.S.C. § 9543). By law, the data provided 
by schools, staff, and students may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as 
required by law (20 U.S.C. § 9573). 
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Exhibit A-3. PIRLS 2016 School Recruitment Letter 


[Date] 

[Title] [Name First] [Name Last], [Title/ Department] 
[School District] 

[Address 1] 

[City], [State] [Zip code] 


Dear [Title] [Name Last]: 


The United States is participating in an important international study in 2016: the Progress in International 
Reading Literacy Study (PIRLS). Since 2001, PIRLS has measured trends in reading achievement at grade 4 in 
countries around the world, including the United States. Results from PIRLS are used by researchers and 
policymakers to chart national progress against international standards and other countries around the world, 
informing national discussions about international competitiveness. 


Your school has been randomly selected to participate in PIRLS in spring 2016. For the first time PIRLS will 
also include an innovative assessment of online reading called ePIRLS. Some classrooms selected to 
patticipate in PIRLS may also be asked to take part in ePIRLS. Participating schools will receive $200, and 
each school’s PIRLS school coordinator (the school staff person designated to work with PIRLS staff) will 
receive $100 as a thank you for his or her time and effort. An additional $50 will be given to the coordinator 
for assisting with ePIRLS. A school administrator and selected teachers will each be asked to complete a 
questionnaire. Teachers will receive $20 as a thank you for completing the questionnaire. Participating 
students will receive a small gift as a token of appreciation. 


PIRLS is described in more detail in the enclosed materials. In the United States, PIRLS is conducted by the 
National Center for Education Statistics (NCES), part of the U.S. Department of Education, and the data are 
being collected by RTI International. The U.S. Office of Management and Budget has approved the data 
collection under OMB #1850-0645. For information on the confidentiality of the data collected, please see 
the enclosed FAQ. While participation in this study is voluntary, your school’s participation is invaluable so 
that the United States has a representative sample of schools across the country. 


Within the next few days, a representative of RTI International will contact you to discuss your participation. 
If you have any questions, please do not hesitate to call Dr. Patricia Green at (XXX) XXX-XXXX or send an 
email to PIRLS@rti.org. You may also get more information about this study by contacting Dr. Sheila 
Thompson at NCES at (202) 502-7425 or sheila thompson@ed.gov, or by visiting the PIRLS website at 
https://nces.ed.gov/surveys/PIRLS /index.asp. 


Your participation in the PIRLS 2016 is very important to its success. Thank you for your time and for 
supporting this important international study. 


Sue Betka Pegey Carr 
Acting Director, Institute of Education Sciences Acting Commissioner, NCES 
Enclosures 


NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002; 20 U.S.C. § 9543). By law, the data provided 
by your school, staff, and students may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except 
as required by law (20 U.S.C. § 9573). 
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Exhibit A-4. PIRLS 2016 School Coordinator Letter 


DATE 


«SCH_ENTITY_NAME» 
«SCH_ADDRESS» 
«sch_citystzip» 


Dear «sch_coord_name», 


We are looking forward to working with you and your school this school year on the Progress in International 
Reading Literacy Study (PIRLS). As the designated school coordinator for PIRLS you play a critical role in 
assisting us with study preparations. We appreciate your assistance, particularly given the many demands on 
your time. 


The attached document, PIRLS School Coordinator Responsibilities, will provide you with an overview of 
PIRLS, as well as a list of tasks and timeline needed to prepare for PIRLS data collection. The first step is to 
provide a list of the 4% grade classes in your school, so that we may select the classes and students who will be 
invited to participate. This step is critically important, and we would appreciate your help in completing this 
task within three weeks of receipt of this letter. 


PIRLS is conducted by the U.S. Department of Education’s National Center for Education Statistics (NCES) 
and data are collected by RTI International (RTI). A PIRLS study representative will contact you shortly to 
answer questions you may have and to begin discussing data collection logistics. In-school data collection is 
scheduled to take place during February through May 2016. The student data collection will include a reading 
assessment and background questionnaire. For the first time PIRLS will also include an innovative 
assessment of online reading called ePIRLS. Some classrooms selected to participate in PIRLS may also be 
asked to take part in ePIRLS. 


RTI will provide a trained test administrator (TA) to conduct the student sessions and to assist with the 
patental permission form process. A school administrator and the teachers of selected classes will also be 
asked to complete a questionnaire. They will be contacted separately, and their data will be collected through 
a web-based application or hard-copy survey. 


We sincerely appreciate your help in preparing for the session at your school and in ensuring that PIRLS is a 
success. Each school’s participation is critical to the success of the study, and reports will not identify 
participating districts, schools, students, or individual staff. 


If you have any questions, please contact us at RTI at (XXX) XXX-XXXX or by email at PIRLS@rti.org. 
Thank you for your support of education through participating in PIRLS. 
Sincerely, 


Patricia Green 
Project Director, Progress in International Reading Literacy Study (PIRLS) 


Enclosures 
NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002; 20 U.S.C. § 9543). By lam, the data 


provided by your school, staff, and students may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other 
purpose except as required by law (20 U.S.C. § 9573). 
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Exhibit A-5. PIRLS School Coordinator Responsibilities 


As the PIRLS school coordinator (SC), you play an important role in the success of the study. RTI 


will provide a trained test administrator (TA) to conduct the student sessions and to assist you 
as much as possible. But we will need to rely on you to complete certain tasks so that data 


collection will be successful. 


Study Details 
PIRLS 2016 consists of the following: 


e Student Reading Assessment and 
Questionnaire -— Students in selected 
classrooms will be administered a reading 
assessment and background questionnaire. 
Some classrooms selected to participate in 
PIRLS may also be asked to take part in a 
new, an innovative assessment of online 
reading called ePIRLS. The main assessment 
will take about 2 hours, and ePIRLS will take 
about 80 minutes. The TA from RTI will 
conduct the student session at your school 
and each participating student will receive a 
thank you gift. 


e School Administrator Questionnaire - The 
school administrator or designee will 
complete a questionnaire about school 
characteristics, | enrollment, resources, 
policies, and the learning environment. The 
questionnaire may be completed online or 
via hardcopy and will take about 40 minutes 
to complete. RTI will send information to you 
to distribute to the person who is designated 
to answer the school administrator 
questionnaire. 


e School Teacher Questionnaire — Teachers of 
selected classrooms will complete a 
questionnaire regarding their teaching 
experience, available resources, and 
instructional practices. The questionnaire 
may be completed online or via hardcopy 
and will take about 40 minutes to complete. 
RTI will send information to you to distribute 
to selected teachers. 
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For additional information, you may visit our 
website at: https://xxx.xxx.xxx 


Your Responsibilities 

You have been asked to complete several tasks 
prior to the student data collection sessions that 
will take place in the spring of 2016. These 
include providing student information, working 
with RTI on the session logistics, assisting with 
parental consent form distribution and receipt, 
and helping to coordinate school staff surveys. 
We will use the PIRLS secure website to exchange 
information, such as class and student lists. 
Please visit: 
https://nces.ed.gov/surveys/PIRLS /index.asp 


and log-in with the following information: 

Study ID: #HHHHHHHHHHH 

Password: #####H## 
Table 1 shows the activities you will be asked to 
perform and a timeline for these activities. 


Table 1. Activities Timeline 
Provide information on | Within 3 weeks of 
4" grade classes using receipt of request 
the Class Listing Form. 
Complete Student Within one week 
Listing Form for of selection of 
selected class(es) classes 
Coordinate session Within 3 weeks of 
logistics (date, time, receipt of request 
location, consent type) 


Notify teachers, At least 3 weeks 
selected students, and prior to scheduled 
parents of the study session 


and benefit of 
participating 
Distribute parental At least 3 weeks 
consent forms to all prior to scheduled 
selected students session 


Monitor return of 
consent forms; 
distribute reminder 
letters as needed. 
Collect returned, signed 
forms and submit to TA. 
Runa systems check of | At least 2 weeks 
the ePIRLS system prior to scheduled 
session 
Notify/remind teachers | One week prior 
and students about the | and one day prior 
data collection sessions | to session 

Assist the TA with On the day of the 
getting the students to | session 

the sessions and setting 
up the room (if 
necessary) 

Coordinate completion | Two weeks prior 
of the administrator to the student 
and teachers’ session 
questionnaires. 


During the 3 
weeks from 
distribution to 
data collection day 


Providing Student Information 

We will select one or two classes per school, 
depending on the number of classes available at 
grade 4. Please submit the Class Listing Form 
(including all classes that account for each 
student in 4" grade). The CLF should be 
uploaded to the secure PIRLS website. The 
student sample will consist of all students in the 
selected classes. Some classrooms selected to 
participate in PIRLS may also be asked to take 
part in ePIRLS. 

We will then ask you to complete the Student 
Listing Form for selected class(es) to provide 
information on the students in those classes 
(e.g., name, district ID, class, date of birth, 
gender). SLF should be uploaded to the secure 
PIRLS website. 

Assisting with Student Session Logistics 

In order to minimize disruption at the school on 
the day of the student data collection, it is crucial 
that arrangements be made in advance. These 
arrangements include: 
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e Determine the date for student data 
collection. 


e Determine the location and time(s) of the 
student session(s): will the student sessions 
take place in the classroom or will a separate 
room be reserved? If there are multiple 
classes sampled, will the administration be 
conducted together or separately? 


e Determine parental permission type 
(implied/passive or written/active). 
Determine the best method of distributing 
consent forms, tracking returned, signed 
forms, collecting returned forms and 
submitting to the PIRLS TA. 


e Work with the TA to identify students with 
special needs. 


e Notify teachers and students in selected 
classrooms about PIRLS, the importance of 
participating, and explain test day activities. 


e Contact parents to encourage them to 
return the signed consent form (if 
applicable). 


Parental Permission 

Parental permission materials will be supplied by 
RTI a few weeks before the scheduled session. 
We can either mail them directly to parents or 
ask that you distribute the parent permission 
forms to the sampled students. 


e Most schools use implied/passive 
permission (which means students only 
return a form if their parent denies 
permission to participate), as this option 
lessens the burden on school staff and 
results in higher participation rates. If you 
are using this permission type, please record 
parent refusals onto the student tracking 
form (STF) which will be provided with the 
permission forms. 


e Some schools’ require  written/active 
consent. If your school requires active 
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consent, parents are asked to return signed 
permission forms to the school coordinator. 
Please keep track of the return of these 
forms on the STF. Please make sure any 
returned forms have one “box” checked, a 
parent/guardian signature (not just a printed 
name) and the name of the student. 


Your TA will be in contact with you to track the 
return of consent forms, as well as exclusion 
statuses for students. Reminder forms will be 
sent home as needed. The TA will check the 
permission forms on the day of the first session 
to make sure we do not include anyone whose 
parents have not granted permission. Please 
keep the returned parent permission forms ina 
locked or secure location. Your TA will fax any 
returned forms from your school at the end of 
the data collection to a secure fax machine at 
RTI. Please let him/her know if you need to retain 
copies. 

Assist on Day of Student Sessions 

The PIRLS TA will arrive at the school about an 
hour before the student session. In order to have 
valid results from the study, we need as many 
sampled students to participate as possible. We 
are depending on you to make certain the 
students and teachers are aware of the date, 
time, and location for their participation and to 
generate enthusiasm. This often makes the 
difference in high student participation. You may 
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want to advertise PIRLS throughout the school 
and classroom or make a PA announcement 
about the study the day before and the day of 
the session to generate interest and encourage 
participation. 

We would also like for you or a teacher to remain 
in the room during the session administration to 
help maintain order and assist as needed. 

Again, the TA will need to fax any returned 
permission forms at the end of the session. 
Please assist him or her in faxing these forms. 
Token of Appreciation 

As a token of our appreciation for your time and 
energy with PIRLS, you will receive $100 after the 
student session has been completed and $50 for 
assistance with ePIRLS. 


THANK YOU for your help to make PIRLS a 
success!! We greatly appreciate your time and 
assistance! 
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Exhibit A-6. PIRLS 2016 Administrator Questionnaire Letter 


[Date] Web Address: 
[Title] [Name First] [Name Last], [Principal/ Administrator] Your Study ID: 
[School] Your Password: 


[Address 1] 
[City], [State] [Zip code] 


Dear [Title] [Name Last]: 


[School Name] is participating in an important international study in 2016: the Progress in International 
Reading Literacy Study (PIRLS). Since 2001, PIRLS has measured trends in reading achievement at grade 4 in 
countries around the world, including the United States. Results from PIRLS are used by researchers and 
policymakers to chart national progress against international standards and other countries around the world, 
informing national discussions about international competitiveness. 


We ate asking you to complete a 40-minute questionnaire to provide insight into the practices and resources 
at your school. The survey is designed to be completed by you or a staff person you designate who can 
provide information about the characteristics of the school, its enrollment, resources, policies, and learning 
environment. The questionnaire may be completed by hardcopy or online using a secure website. To access 
the questionnaire online, please use the web address and unique study ID and password provided on this 
letter. If completing the questionnaire by hardcopy, please use the enclosed business reply envelope to return 
the completed questionnaire to us. You may also give the completed hard copy questionnaire to the PIRLS 
school coordinator at your school. 


In the United States, PIRLS is conducted by the National Center for Education Statistics (NCES), part of the 
U.S. Department of Education, and the data are being collected by RTI International. The U.S. Office of 
Management and Budget has approved the data collection under OMB #1850-0645. For information on the 
confidentiality of the data collected, please see the enclosed FAQ. While participation in this questionnaire is 
voluntary, your participation is invaluable. 


If you have any questions, please do not hesitate to call Dr. Patricia Green at (XXX) XXX-XXXX or send an 
email to PIRLS@rti.ore. You may also get more information about this study by visiting the PIRLS website at 


https://nces.ed.gov/ sutveys/PIRLS /index.asp. If you have questions about your rights as a study participant, 


you may call RTI’s Office for Research Protection in Durham, North Carolina, toll-free at 1-866-214-2043. 


Your participation in the administrator questionnaire for PIRLS 2016 is very important to its success. Thank 
you for your time and for supporting this important international study. 


Sue Betka Pegey Carr 
Acting Director, Institute of Education Sciences Acting Commissioner, NCES 
Enclosures 


NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002; 20 U.S.C. § 9543). By law, the data provided 
by you may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 
U.S.C. § 9573). 
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Exhibit A-7. PIRLS 2016 Teacher Questionnaire Letter 


[Date] Web Addtess: 
[Title] [Name First] [Name Last], [Teacher] Your Study ID: 
[School] Your Password: 


[Address 1] 
[City], [State] [Zip code] 


Dear [Title] [Name Last]: 


[School Name] is participating in an important international study in 2016: the Progress in International 
Reading Literacy Study (PIRLS). Since 2001, PIRLS has measured trends in reading achievement at grade 4 in 
countries around the world, including the United States. Results from PIRLS are used by researchers and 
policymakers to chart national progress against international standards and other countries around the world, 
informing national discussions about international competitiveness. 


We are asking you to complete a 40-minute questionnaire to provide insight into your teaching experience 
and available resources and instructional practices at your school. You may complete the questionnaire by 
hardcopy or online using a secure website. To access the questionnaire online, please use the web address and 
unique study ID and password provided on this letter. If completing the questionnaire by hardcopy, please 
use the enclosed business reply envelope to return the completed questionnaire to us. You may also give the 
completed hard copy questionnaire to the PIRLS school coordinator at your school. 


You will receive a check for $20 within a few weeks of completing the questionnaire as a token of our 
appreciation. 


In the United States, PIRLS is conducted by the National Center for Education Statistics (NCES), part of the 
U.S. Department of Education, and the data are being collected by RTI International. The U.S. Office of 
Management and Budget has approved the data collection under OMB #1850-0645. For information on the 
confidentiality of the data collected, please see the enclosed FAQ. While participation in this questionnaire is 
voluntary, your participation is invaluable. 


If you have any questions, please do not hesitate to call Dr. Patricia Green at (XXX) XXX-XXXX or send an 
email to PIRLS@rti.ore. You may also get more information about this study by visiting the PIRLS website at 
https://nces.ed.gov/surveys/PIRLS/index.asp. If you have questions about your rights as a study participant, 
you may call RTT’s Office for Research Protection in Durham, North Carolina, toll-free at 1-866-214-2043. 


Your participation in the teacher questionnaire for PIRLS 2016 is very important to its success. Thank you 
for your time and for supporting this important international study. 


Sue Betka Peggy Carr 
Acting Director, Institute of Education Sciences Acting Commissioner, NCES 
Enclosures 


NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002; 20 U.S.C. § 9543). By law, the data provided 
by you may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 
U.S.C. § 9573). 
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Exhibit A-8. PIRLS 2016 Implicit Parent Permission 


U.S. DEPARTMENT OF EDUCATION 
INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CENTER FOR EDUCATION STATISTICS 


Dear Parent or Guardian, 


This letter is to inform you about an important international study of student learning being conducted in our 
school this spring. The Progress in International Reading Literacy Study (PIRLS) provides important 
information for benchmarking student performance in reading in the United States against countries around 
the world. Since 2001, PIRLS has measured worldwide trends in student reading skills at grade 4. The next 
PIRLS assessment will be in the spring of 2016. 


Your child’s school has accepted an invitation from the National Center for Education Statistics (NCES), part 
of the U.S. Department of Education, to participate in PIRLS. One 4th grade classes will take part. This is 
your child’s class. The enclosed summary sheet provides some background information about PIRLS, 
explains what is involved for each student selected to participate in the study, and gives a contact phone 
number and email address where you can find answers to any questions you might have. 


To have an accurate picture of what U.S. 4th graders can do in reading, it is important that each student 
selected take part in the study. In addition to answering reading questions, students will be asked to complete 
a brief questionnaire about themselves. For the first time PIRLS will also include an innovative assessment of 
online reading called ePIRLS. I urge you to support this effort by encouraging your child to take part; 
however, participation in this study is entirely voluntary. Previous experience suggests that students actually 
enjoy taking part, and participating students will receive a small gift, which we think they will like. 


All of the information collected is safeguarded, as required by law. NCES is authorized to conduct this study 
under the Education Sciences Reform Act of 2002 (ESRA, 2002; 20 U.S.C. § 9543). Under that law, the data 
provided by schools, staff, and students may be used only for statistical purposes and may not be disclosed, 
ot used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). Students and 
schools are never identified in any reports. All reported statistics refer to the United States as a whole. 


If you have any objection to your child joining in the PIRLS activities, please let us know by completing the 
attached consent form and returning it to the school. 


Por more information, please visit the study website at https://nces.ed.gov/surveys/PIRLS/index.asp. You 
may also call the study hotline at (866) 800-9176 or send email to pirls@rti.org. Thank you for taking the time 
to learn about this important study. 


Sincerely, 


Peggy G. Carr, Ph.D. 
Acting Commissioner, NCES 


\ 
y 


Enclosures: 
Pacts for Parents about the PIRLS 2016 Main Study 
Parent/Guardian Consent Form 
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Progress in International Reading Literacy Study (PIRLS) 


Parent/Guardian Consent Form 


Your child has been asked to participate in an international study of student learning called the 
Progress in International Reading Literacy Study (PIRLS). Each student who participates will recetve 
a small gift. 


The student assessment will be administered by a team of researchers from RTI International, on 
behalf of the National Center for Education Statistics (NCES). 


If you grant permission for your child to participate in PIRLS, you do not need to return this 
form. 


If you do not consent to your child’s participation in PIRLS, please return this form to your 


child’s school as soon as possible. 


I do not grant permission for my child, , to participate in 


the Progress in International Reading Literacy Study. 


(Signature of parent or guardian) 


Date of signature: / }- 


PLEASE PRINT: 


Student name: 


School name: 


FOR OFFICE USE ONLY: 


Student ID: 


«StudentFName» 


«StudentID» 
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Exhibit A-9. PIRLS 2016 Explicit Parent Permission 


U.S. DEPARTMENT OF EDUCATION 
INSTITUTE OF EDUCATION SCIENCES 


NATIONAL CENTER FOR EDUCATION STATISTICS 


Dear Parent or Guardian, 


This letter is to inform you about an important international study of student learning being conducted in our 
school this spring. The Progress in International Reading Literacy Study (PIRLS) provides important 
information for benchmarking student performance in reading in the United States against countries around 
the world. Since 2001, PIRLS has measured worldwide trends in student reading skills at grade 4. The next 
PIRLS assessment will be in the spring of 2016. 


Your child’s school has accepted an invitation from the National Center for Education Statistics (NCES), part 
of the U.S. Department of Education, to participate in PIRLS. One 4th grade class will take part. This is your 
child’s class. The enclosed summary sheet provides some background information about PIRLS, explains 
what is involved for each student selected to participate in the study, and gives a contact phone number and 
email address where you can find answers to any questions you might have. 


To have an accurate picture of what U.S. 4th graders can do in reading, it is important that each student 
selected take part in the study. In addition to answering reading questions, students will be asked to complete 
a brief questionnaire about themselves. For the first time PIRLS will also include an innovative assessment of 
online reading called ePIRLS. I urge you to support this effort by encouraging your child to take part; 
however, participation in this study is entirely voluntary. Previous experience suggests that students actually 
enjoy taking part, and participating students will receive a small gift, which we think they will like. 


All of the information collected is safeguarded, as required by law. NCES is authorized to conduct this study 
under the Education Sciences Reform Act of 2002 (ESRA, 2002; 20 U.S.C. § 9543). Under that law, the data 
provided by schools, staff, and students may be used only for statistical purposes and may not be disclosed, 
or used, in identifiable form for any other purpose except as required by law (20 USS.C., § 9573). Students and 
schools are never identified in any reports. All reported statistics refer to the United States as a whole. 


Before we can allow your child to join in the PIRLS activities, we must have your written consent. 
Please let us know by completing the attached form and returning it to the school. 


For more information, please visit the study website at https://surveys.nces.ed.gov/pirls/index.asp. You may 
also call the study hotline at (866) 800-9176 or send email to pirls@rti.org. Thank you for taking the time to 
learn about this important study and to consider your child’s participation in it. 


Sincerely, 


| 
y 

Feanyy \ 

Tea KS an 


Pegey G. Carr, Ph.D. 

Acting Commissioner, NCES 

Enclosures: 

Pacts for Parents about the PIRLS 2016 Main Study 
Parent/Guardian Consent Form 
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Progress in International Reading Literacy Study (PIRLS) 
Parent/Guardian Consent Form 


Your child has been asked to participate in an international study of student learning called the 
Progress in International Reading Literacy Study (PIRLS). Each student who participates will receive 
a small gift. 


The student assessment will be administered by a team of researchers from RTI International, on 
behalf of the National Center for Education Statistics (NCES). 


Please check only one option below to indicate your decision about your child’s 


participation in the study and return this form to your child’s teacher as soon as possible. 
L] Yes, I grant permission for my child to participate in PIRLS. 


z= No, I do not grant permission for my child to participate in PIRLS. 


(Signature of parent or guardian) 


Date of signature: / / 


PLEASE PRINT: 


Student name: 


School name: 


FOR OFFICE USE ONLY: 


Student ID: 


«StudentFName» 


«StudentID» 
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Exhibit A-10.PIRLS 2016 Study Brochure 


Other information collected by PIRLS 


PIRLS is more than an assesment of student 
knowledge and skills in reading. PIRLS also 
considers the context in which learning occurs. 
Students, teachers, and schools are asked 
about a variety of aspects of the environments 
in which content is taught, learned, practiced, 
and applied. In this way, PIRLS provides each 
country with arich source of information onthe 
factors influencing reading achievement. 


Countries that participated in PIRLS 2011 


Argentina Italy 

Australia Kuwait 

Austria Latvia 

Azerbaijan Lithuania 

Bahrain Luxembourg 
Belgium Macedonia 

Belize Malta 

Botswana Moldova 

Bulgaria Morocco 

Canada The Netherlands 
Chinese Taipei New Zealand 
Colombia Norway 

Croatia Poland 

Cyprus Portugal 

Czech Republic Qatar 

Denmark Russian Federation 
Finland Saudi Arabia 
France Singapore 
Georgia Slovak Republic 
Germany Slovenia 

Hong Kong-China South Africa 
Hungary Spain 

Iceland Sweden 

Indonesia Trinidad & Tobago 
Iran Turkey 

Ireland United Arab Emirates 
Israel United States 


Results 


Results from PIRLS 2016 will be available in Decem- 
ber 2017, and examples of released PIRLS items 
are available at http://nces.ed.gov/surveys/pirls/ 
released.asp. 


Findings from PIRLS 2011 


e The overall reading average scale score for U.S. 
students (556) was higher than the international 
PIRLS scale average, which is set to 500. 


e The United States average was higher than 40 
education systems and not measurably different 
from 7 others. Five education systems had higher 
averages. 


e The average score for girls was higher than the 
average score for boys in the United States (562 
vs. 551). 


NCES is authorized to conduct PIRLS under the Education Sci- 
ences Reform Act of 2002 (ESRA 2002), 20 U.S. Code, § 9543. In- 
formation collected will help the U.S. Department of Education's 
ongoing efforts to benchmark student achievement in the United 
States. Participation is voluntary. By law, data collected may be 
used only for statistical purposes and may not be disclosed, or 
used, in identifiable form for any other purpose except as re- 
quired by law (20 U.S. Code, § 9573). The U.S. Office of Manage- 
ment and Budget has approved the data collection under OMB # 
1850-0645. Individual responses will be combined with those from 
other participants to produce summary statistics and reports. 


For questions about PIRLS 2016, contact the 
PIRLS Information Hotline at 1-866-800-9176 
or email PIRLS@rti.org. 


e 
e 
e 1es NATIONAL CENTER for 
. EDUCATION STATISTICS 
Institute of s 


Educotion Sciences 
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Progress in International Reading 
Literacy Study (PIRLS) 2016 
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What is PIRLS? 


The Progress in International Reading Literacy 
Study (PIRLS)is an international assessment and 
research project designed to measure reading 
achievement at the fourth grade, as well as 
school and teacher practices related to reading 
instruction. PIRLS was assessed in 2001, 2006, 
and 2011, with the United States participating 
in all past assessments. In 2016, PIRLSwill in- 
volve students from more than 40 countries, 
including the United States. For the first time 
PIRLSwill also include an innovative assessment 
of online reading calledePIRLS. 


PIRLS is sponsored by the International As- 
sociation for the Evaluation of Educational 
Achievement (IEA) and conducted in the Unit- 
ed States by the National Center for Education 
Statistics (NCES), part of the U.S. Department 
of Education. 


Why is PIRLS important? 


PIRLS provides a unique opportunity to com- 
pare the reading, knowledge, and skills of U.S. 
fourth-grade students with that of their peers 

in countries around theworld. PIRLS comple- 
ments what we learn from national assessments 
by identifying the strengths and weaknesses of 
student performance relative to students around 
the world. The results inform national discus- 
sions about education as well as international 
competitiveness. 


PIRLS provides valuable benchmark informa- 
tion on how U.S. students compare to students 
around the world, allows educators and poli- 
cymakers to examine other education systems 
for practices that could have application to 

the United States, and contributes to ongoing 
discussions of ways to improve the quality of 
education for allstudents. 


Moreover, by participating in PIRLS 2016, the 
United States will obtain data about changes 
in children’s reading achievement over the past 
15 years, including valuable information about 
changes in reading instruction, how those 
changes relate to students’ performance in 
reading, and about home, school, and class- 
room influences on readingachievement. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


What type of assessment is PIRLS? 


PIRLSis developed through an international 
consensus-building process involving input 
from U.S. and international experts in reading 
and measurement. The assessment is carefully 
constructed to assess a range of reading 
comprehension strategies for twomajor reading 
purposes using informational text and literary 
text. The assessment includes reading passages 
ife)|(e\WVZ=10l oNYae) oX-1abr=yare(=\o m= lace manlUliiie)(=sre1ale)(ex-) 
format questions about the text. The student 
questionnaire gathers information about the 
contexts in which children learn to read and 
children’s attitudes toward reading. Examples of 
released PIRLSitems are available at http:// 
nces.ed.gov/surveys/pirls/released.asp. 


As anew extension to PIRLS in 2016, ePIRLS 
— an assessment of online reading — makesit 
possible for education systems to understand 
how successful they are in preparing fourth 
grade students to read, comprehend, and in- 
terpret online information. 
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Exhibit A-11.PIRLS 2016 Main Study FAQ 


Progress in International Reading Literacy Study 
Frequently Asked Questions 
PIRLS 2016 Main Study (Spring 2016) 


What is PIRLS? 


The Progress in International Reading Literacy Study (PIRLS) is an international assessment and research 
project designed to measure trends in reading achievement at the fourth-grade level as well as school 
and teacher practices related to instruction. Since 2001, PIRLS has been administered every 5 years. 
PIRLS 2016, the fourth study in the series, will involve students from more than 40 countries, including 
the United States. For the first time PIRLS will include an innovative assessment of online reading called 
ePIRLS. 


Why was my school selected for participation? 


Schools of varying demographics and locations were randomly selected so that the overall U.S. sample is 
representative of the overall U.S. school population. The random selection process is important for 
ensuring that a country’s sample accurately reflects its schools and, therefore, can be compared fairly 
with samples of schools from other countries. 


Will all our fourth-grade students be asked to participate? 


It depends on the number of fourth-grade classrooms in the school. In schools with only one or two such 
classrooms, all students will be asked to participate. In schools with more than two such classrooms, 
only students in two randomly selected classrooms will be asked to participate. Some classrooms 
selected to participate in PIRLS may also be asked to take part in ePIRLS. In addition, some students with 
special needs or limited English proficiency may be excused from the assessment. 


Who conducts the PIRLS assessment? 


The entire assessment process will be conducted by trained staff from RTI International, a research 
organization under contract with the National Center for Education Statistics in the U.S. Department of 
Education. NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 
(ESRA 2002; 20 U.S.C. § 9543) and approval of the U.S. Office of Management and Budget under OMB # 
1850-0645. 


How are the teacher and school questionnaires administered? 


The teacher and school questionnaires are administered either online from a secure website or via a 
hardcopy form. Teacher questionnaires take about 40 minutes to complete and ask teachers questions 
about their experience, available resources, and instructional practices. School questionnaires take 
about 40 minutes to complete and ask about school practices and resources. 


Do teachers need to help administer the assessment? 


No, RTI International field staff will visit the school on the day of the assessment, bringing with 
them all the materials required. These field staff will administer the assessments to students. 


When will the assessment be conducted? 


The assessment will be conducted between February and May, 2016. RTI International will work with 
schools to identify an assessment date convenient for the school in that time period. 
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Where will the assessment be conducted? 
The assessment will be conducted in the schools that are selected to participate. 
How long does the assessment take? 


The main assessment session is approximately 2 hours, including time for directions. The ePIRLS session 
will take 80 minutes and will include breaks between sections. 


What will happen with the collected data? 


The data from the assessment will be used to evaluate how the knowledge and skills of U.S. students 
compare to those of their peers in other participating countries. By law, the data provided by schools, 
staff, and students may be used only for statistical purposes and may not be disclosed or used, in 
identifiable form for any other purpose except as required by law (Education Science Reform Act of 2002 
[ESRA 2002; 20 U.S.C. § 9573]). Reports of the findings from the assessment will not identify 
participating districts, schools, students, or individual staff. Individual responses will be combined with 
those of other participants to produce summary statistics and reports. 


Are schools required by federal law to participate? 


No. School participation is voluntary. However, we hope you will participate in this study so that 
students like those in your school are accurately and fairly represented. 


What are the benefits? 


The nation as a whole benefits from PIRLS by having a greater understanding of how the 
reading knowledge and skills of U.S. 4'* graders compare with 4" graders from other countries. 
To thank participating schools and individuals for their time and effort we offer tokens of 
appreciation: schools that participate in PIRLS will receive $200 and the school coordinator 
(staff person designated to assist with the study) will receive $100. An additional $50 will be 
given to the coordinator for assisting with ePIRLS. Teachers who complete a survey will receive 
$20, and each student who participates will receive a small gift. 


Where can | find more information about PIRLS? 
Visit the PIRLS website at https://nces.ed.gov/surveys/pirls/index.asp. 


For additional information about PIRLS 2016, contact the PIRLS U.S. home office at (XXX) XXX-XXXX or 
email PIRLS@rti.org. 
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Exhibit A-12. Summary of School Activities: PIRLS 2016 Main Study 


Summary of School Activities: PIRLS 2016 Main Study 


February-May 


November- January-March 2016 2016 
Questionnaire | December 2015 Prior to assessment day Assessment day Benefits 
Principal " Designate a Complete a brief School = Support Represent 
school Questionnaire on the characteristics assessment day other similar 
coordinator of the school, its enrollment, activities U.S. schools 
resources, policies, and learning = Encourage Receive a $200 
environment (available online or students to check for the 
hardcopy) participate and school 
Share the importance of do their best on 
participation in the study with the assessment 
school staff 
School = Select an Arrange the day and location for the | = Confirm space Receive a $100 
coordinator assessment date PIRLS and ePIRLS sessions. for assessment check 


Teachers of 
sampled 
classes 


convenient for 
your school 


Confirm dates and location with 
PIRLS assessment staff 

Provide class lists, student lists, and 
contact information for grade 4 
students 

Notify teachers, selected students, 
and students’ parents of the study 
and benefit of participating 

Work with assessment staff to 
identify students with special needs 
Coordinate the principal’s 
completion of the School 
Questionnaire (online or hardcopy) 
Coordinate the teacher’s 
completion of the Teacher 
Questionnaire (online or hardcopy) 
Collect parental consent forms 
where required and submit to PIRLS 
staff 

Run a systems check of the ePIRLS 
system 

Complete Teacher Questionnaire 
and return to school coordinator 
prior to assessment day (if not 
completed online) 


is problem-free 

= Collect complete 
School and 
Teacher 
Questionnaires 
(if not 
completed 
online) and give 
to assessment 
staff 

= Ensure all 
sampled 
students attend 
the assessment 
session 

= Meet with 
assessment staff 
and provide 
feedback about 
the assessment 
process 


Receive an 
additional $50 
for running the 
ePIRLS system 
check, and 
assisting with 
computer 
setup. These 
components 
may be 
delegated toa 
school IT 
coordinator if 
necessary. 


Receive a $20 
check 
Represent the 
United States in 
preparations 
for the 
international 
study 


Selected 
Students 


= Attend the 
assessment 
session, 
complete the 
assessment and 
Student 
Questionnaire 


Receive a small 
thank-you gift 
Represent the 
United States in 
preparations 
for the 
international 
study 
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Summary of School Activities: PIRLS 2016 Main Study 


February-May 


coordinator at 
the end of the 
assessment 

= Maintain 
security of all 
materials 


November- January-March 2016 2016 
Questionnaire | December 2015 Prior to assessment day Assessment day Benefits 
RTI =" Work with the = Call the school coordinator to discuss | ™ Conduct Ensure quality 
International school to set an assessment day location(s) and assessment and uniformity 
assessment assessment student participation from start to of data 
staff date = Select classroom sample and notify finish collected across 
= Help school school of selected classes = Furnish all the the United 
coordinator = Provide School and Teacher assessment States 
with Questionnaires to the school materials, 
assessment coordinator for distribution pencils, and test 
details and booklets 
logistics =" Conduct a brief 
= Protect school debriefing 
and student interview with 
confidentiality the school 


For additional information, go to https://nces.ed.gov/surveys /pirls /index.asp. 
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Appendix B. Field Staff Training Materials 


Contents 


Exhibit B-1. Test Administrator Training Agenda 
Exhibit B-2. PIRLS Session Script 
Exhibit B-3. ePIRLS Session Script 


Exhibit B-4. Test Administration Form 
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Exhibit B-1. Test Administrator Training Agenda 


Day 1 


PIRLS/ePIRLS 


Test Administrator Training Agenda 


Registration, Fingerprinting, Confidentiality Forms/Affidavits (120) 


Module 1.0 


Module 1.1 
Module 1.2 
Module 1.3 


Module 1.4 


Module 1.5 
Module 1.6 
Break (15) 
Module 1.7 


Module 1.8 


Welcome and Introduction (15) 

Break (15) 

Purpose and Background of PIRLS (15) 

Recruitment and Sample Selection (5) 

Overview of TA’s Responsibilities (10) 

Permission Types and Permission Form Tracking (30) 
Lunch (75) 

Case Assignment Materials (75) 


Working with School Coordinator (75) 


Test Administration Logistics (45) 


Test Security (15) 
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Day 2 

Module 2.0 
Module 2.1 
Module 2.2 


Module 2.3 


Module 2.4 


Module 2.5 
Module 2.6 


Module 2.7 


Module 2.8 
Module 2.9 
Module 2.10 


Module 2.11 
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Day 1 Recap (15) 

Student Assessment (15) 

Student Questionnaire (5) 

Test Administration Mock (120) 
Break (15) 

Make-Up Days and Response Rates (60) 
Lunch (75) 

Dealing with Problems (15) 

Staff Questionnaires (15) 

Contacting Parents (15) 

Break (15) 

Project Laptop Setup with FTSG (60) 
Overview of ePIRLS (15) 

ePIRLS Computer Requirements (30) 


Testing ePIRLS (70) 
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Day 3 
Module 3.0 
Module 3.1 


Module 3.2 


Module 3.3 


Module 3.4 
Module 3.5 
Module 3.6 


Module 3.7 


Module 3.8 
Module 3.9 


Module 3.10 


Day 2 Recap (15) 

Setting up school computers (10) 
Administering ePIRLS (with demo) 

Break (15) 

Setting up laptops with practice (120) 

Lunch (75) 

Administering ePIRLS (with demo) cont. (75) 
After the ePIRLS Session (45) 

Phone report to FS (15) 

Entering STF and TAF datain the FRS (120) 
Break 

Reviewing reports in the FRS (15) 

Quality control Measures (10) 


Certification Overview (10) 
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Day 4 
Module 4.0 
Module 4.1 


Module 4.2 


Module 4.3 
Module 4.4 


Module 4.5 
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Day 3 Recap (15) 

Administrative Procedures (75) 
PIRLS/ePIRLS review/Q&A session (30) 
Working Lunch/Certification setup (30) 
Certification (180) 

Training and Evaluation Form (10) 


Distribution of Assignments (10) 
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Exhibit B-2. PIRLS Session Script 


The instructions marked with the LL) symbol and printed in bold font style in the administration 
script must be read aloud to the students word for word to ensure that the testing sessions are 
conducted in the same way in all countries. Although you should become familiar with these 
instructions before the actual testing, do not attempt to memorize them. Read these instructions 
exactly as they are written. Comments that are not in bold are not to be read aloud. They are 
instructions for you only. 


To begin the testing session: 
[/ Make sure that the Class ID is recorded at the top of the Test Administration Form. If 


missing, you can find it on the Student Tracking Form, Column [b]. 


[] Make sure the students are seated quietly, with nothing on the desk except for a pen 
or pencil. 


// Record the current time in Cell (8a) of the Test Administration Form. 
[/ Begin reading the Administration Script. 
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Exhibit B-2. PIRLS Session Script—Continued 


The Administration Script 


LY This school has been chosen to take part in an important international project to 
study how well children around the world can read. Different countries from all 
over the world are taking part in this study. While I talk to you about today’s test, I 
would like you all to be quiet, stay at your desks, and listen carefully. 


LY Now I will pass out the test packets. Do not open them until I tell you to do so. Each 
of you will receive one test packet. Not all packets are the same. 


LY If you still have any schoolbooks or papers on your desk, please put them away. All 
electronic devices, such as cell phones, portable computers, photo or video cameras, 
must be stored away for the duration of the test administration. 


As you hand out the test packets, make sure that each student receives the packets specially 
prepared for him or her. Student initials and ID numbers will be printed on the id label on the 
packets that contain the test booklets. Test Administrators will need to match these to the names 
on the STF to distribute the packets. You will have made sure that the student identification code 
on the test booklet is the same as the corresponding code in column (2) on the Student Tracking 
Form ahead of time, but always ask students to look at the id label on their test booklets and 
verify that their initials are correct. Please make sure that all students who receive PIRLS 
Booklet R also receive the PIRLS Reader. Do not allow students to open the test booklets until 
you tell them to. Record the students’ participation status in Column 8 (Achievement Session) of 
the Student Tracking Form. If you are administering a makeup session, then use the shaded part 
of Column 8 (Achievement Session). 


If a student is absent, put that test packet aside. Do not give it to anyone else, since each test 
booklet is marked for a specific student. 


If there is a student in the classroom who is not listed on the Student Tracking Form, or an 
originally assigned booklet is damaged, give one of the three spare booklets to this student. Make 
sure that the student identification code on the test booklet matches the code for the unused 
record in column (2) on the Student Tracking Form. Once you have matched the codes on the 
booklet and the Student Tracking Form, enter the name, date of birth, and gender of the 
additional student in columns (1), (5) and (6) on the Student Tracking Form. Then, record the 
student’s name (or ID) on the test booklet and on the Student Questionnaire. 


After the test packets are passed out and the Student Tracking Form has been completed, say the 
following to the students: 


LY Turn to the first page in the booklet that says “Directions.” Please read along as I 
read the Directions aloud. 
Directions 


LO In this test, you will read stories or articles and answer questions about what you have 
read. You may find some parts easy and you may find some parts difficult. 
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Exhibit B-2. PIRLS Session Script—Continued 


LY You will be asked to answer different types of questions. Some of the questions will be 
followed by four choices. You will choose the best answer and fill in the circle next to 
that answer. Example I shows this type of question. 


Example 1 


sa] 
1. How many days are there in a week? 


(@) 2 days 
4 days 
© Tdays 


(©) 10 days 


The circle next to “7 days” is filled in because there are 7 days in a week. 


B 


co If you are not sure about the answer to a question, fill in the circle next to the answer 
you think is best, and move on to the next question. 


an) If you decide to change an answer to a question, draw an “X” through your first 
answer like the picture in your booklet. Then fill in the circle next to your new answer. 
Example 2 shows how to do this. 


Example 2 


Pe 
1. How many days are there in a week? 


x 2 days 
4 days 


@ 7 days 
(©) 10 days 


ina As you work through the questions, you will need to look back at the passages you 
have read to help you answer the questions. 


LQ For some questions you will be asked to write your answers in the space provided in 
your booklet. Example 3 shows one question like this. 
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Exhibit B-2. PIRLS Session Script—Continued 
Example 3 


3. Where does the little boy go after he finds the book? 


LY Example 3 has a pencil with a 1 next to it. This means the question is worth 1 point. 
For questions worth 1 point, you need to write a few words, or a sentence. 


LY Example 4 shows a question with a pencil with a 3 next to it. This means the question 
is worth 3 points. For questions worth 2 or 3 points, you need to explain your view 
using what you have read in the story or article. You may write in full sentences if 
you wish. 


Example 4 


4. What makes the ending of the story both happy and sad? Use what 
you have read in the story to help you explain. 


LQ You will have 40 minutes to work in your test booklet and then we will take a short 
break. Then, you will work for another 40 minutes. 


LY Do your best to answer all the questions. If you cannot answer a question, move on to 
the next one. 


Part One 


LY At the end of the first part of your booklet, you will see a STOP message. Do not 
continue with the rest of the booklet until you are told to do so. You can review what 
you just worked on until the time is up or take a book you have with you and read 
quietly at your desk. Do you have any questions? 


When all problems are resolved and you have the students’ attention again, record the current 
time in Cell (8b) of the Test Administration Form. Then say: 


LY Turn the page and begin reading the first story in your booklet. 
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Exhibit B-2. PIRLS Session Script—Continued 


Begin timing the 40 minutes for Part 1. Record the current time in Cell (9a) of the Test 
Administration Form. 


Make sure all students begin working on the correct part of the booklet. Note that students 
assigned the PIRLS Reader and Booklet R should record their responses in Booklet R. They 
should begin by reading the first story in the PIRLS Reader. All other students should begin 
reading the story/article in the first section of their test booklet. These students record their 
answers directly in the booklet. Remember that you are not allowed to help the students with the 
test. While the students are working, you should move around the room to see that students are 
working on the correct section of their booklets. 


If all students are finished at any time after 30 minutes, you may end the testing session at this 
time. 


After 35 minutes have passed, say the following: 


ima You have 5 minutes left to work on this part of your booklet. Make sure you try to 
finish answering all of the questions in the first part of your booklet before the 
break. 


After the last 5 minutes have passed, say: 


LO Your time is up. Please close your booklets, and put your pens or pencils down. Do 
not write anything more. We will now take a 5 minute break. 


Record the current time in Cell (9b) of the Test Administration Form. 


Break 


If the room will be left unattended during the break, collect the booklets from the students one by 
one. Keep the booklets secure during the break time. You will then redistribute the booklets after 
the break just like you did at the beginning of the testing session, making sure each student 
receives the same test booklet he/she was working on during the first half of the testing session. 
Part Two 


Record the current time in Cell (10a) of the Test Administration Form. 


Make sure all the students are seated. When the students are seated and quiet redistribute the test 
booklets, if necessary. Then, say the following: 


(aa Does everybody have his or her test booklet? 


When all problems are resolved and you have the students’ attention again, record the current 
time in Cell (10b) of the Test Administration Form and say: 


LY You are now going to work on the second part of your test booklet. 


LQ At the end of the second part of your booklet, you will see a STOP message. If you 
are finished early, you can review your work on the second part of your booklet. 
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Exhibit B-2. PIRLS Session Script—Continued 


LQ Open your booklet and turn to Part 2. You may begin working. 


Begin timing the 40 minutes for the second part of the session. Record the current time in Cell 
(11a) of the Test Administration Form. 


Make sure all students begin working on the correct part of the booklet. Students assigned the 
PIRLS Reader and Booklet R should be reading the second story in the PIRLS Reader. All other 
students should begin reading the story/article in the second section of the test booklet. 
Remember that you are not allowed to help the students with the test. While the students are 
working, you should move around the room to see that students are working on the correct 
section of their booklets. If a student is finished early, make sure that he or she has a book to 
read. 


If all students are finished at any time after 30 minutes, you may end the testing session at this 
time. 


After 35 minutes have passed, say the following: 


ima You have 5 minutes left to work on this part of your booklet. Make sure you try to 
finish answering all of the questions in the second part of your booklet. 


After the last 5 minutes have passed, say: 
LQ Your time is up, please stop working. 
Record the current time in Cell (11b) of the Test Administration Form and say: 


ian Now please turn to the last page of your booklet and answer the questions about 
how much you liked the things you read. For example, if you liked the story a lot, 
then fill in the circle next to the smiling face and the words “I liked it a lot.” 


Make sure that all students are answering the questions on the back page of the booklet only. If 
they need help answering these questions, you are free to help them. Once they are finished, 
please say: 


LY Thank you for your work. Please stay seated and place your booklets in your 
envelope 


If the Student Questionnaire is going to be administered now, say: 


We will now take a 5 break. Afterwards I will ask you to answer a short 
questionnaire. 
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Exhibit B-2. PIRLS Session Script—Continued 


Administering the Student Questionnaire 


The instructions marked with the LL] symbol and printed in bold font style in this section must 
be read aloud to the students. Comments that are not in bold are not to be read aloud. They are 
instructions for you only. 


Before you begin the questionnaire administration: 


[] Make sure that you have the corresponding Student Tracking Form and Test 
Administration Form. 
[] Make sure all the students are in the class, are seated quietly, and have pencils to 
use. 
When ready, say: 


ina Now please take out the questionnaire from your packet. Do not open the 
questionnaire until I tell you to do so. 


As students take out their questionnaires, make sure that each student received the questionnaire 
specially prepared for him or her. You can do that by asking each student to verify their initials 
on the questionnaire label is correct. Record the student’s participation status in Column 8 
(Questionnaire Session) of the Student Tracking Form. If you are administering a makeup 
session, then use the shaded part of Column 8 (Questionnaire Session). 


If a student is absent, put that questionnaire aside. Do not give it to anyone else since each 
questionnaire is assigned to a specific student. 


After the Student Tracking Form has been completed, say the following to the students: 
(aa Does everybody have his or her questionnaire? 
If yes, then continue. If not, find out why and proceed as described before. 


LO The directions are printed at the beginning of your questionnaire. I will also read 
them to you. It is important that you follow the directions very carefully so that you 
understand how to mark your answers. Now open the questionnaire and turn to the 
first page titled “Directions.” 


Make sure that the students have their questionnaires open to the Directions page before 
proceeding. 


LY Please follow the directions in your questionnaire as I read them aloud. 


Directions 


ian In this booklet, you will find questions about you and what you think. For each 
question, you should choose the answer you think is best. 


LO Let us take a few minutes to practice the kinds of questions you will answer in this 
booklet. 


LO Example 1 is one kind of question you will find in this booklet. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


APPENDIX B. FIELD STAFF TRAINING MATERIALS | B-13 | 


Exhibit B-2. PIRLS Session Script—Continued 


Make sure that all students are following along and are looking at Example 1 in their 
questionnaires. 


Example 1 


Do you go to school? 


Fill one circle only. 


LO In Example 1, the question asks, “Do you go to school?” Below this question are a 
“Yes” and a “No.” Since you all go to school, you should all fill in the circle next to 
“Yes.” 


Give students time to fill in the circle next to “Yes” and make sure they understand how to do it. 
Once everyone has completed the example, move on to Example 2. 


Make sure that all students are following along and are looking at Example 2 in their 
questionnaires. 


LY Example 2 is another kind of question you will find in this booklet. 


Example 2 


How often do you do these things? 


Fill one circle for each line. 


Every day Onceor Once or Never or 
or almost twice a twice a almost 
every day week month never 

a) I talk with my friends ------------------ O O O O 

b) I play sports ------------------------------- O O O O 

c) Iride a skateboard ---------------------- O O O O 


LY This question asks “How often do you do these things?” Letter (a) says, “I talk with 
my friends.” You are given four choices for how often you do this: Every day or 
almost every day; Once or twice a week; Once or twice a month; and Never or 
almost never. 

LQ Fill in the circle below your answer. For example, if you talk to your friends every 
day or almost every day, fill in the first circle under “Every day or almost every 
day.” 
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Exhibit B-2. PIRLS Session Script—Continued 


Give students time to fill in their answers to all parts of the Example 2 question and make sure 
they understand how to answer this kind of question. Once everyone has completed the example, 
move on to Example 3. 


Make sure that all students are following along and are looking at Example 3 in their 
questionnaires. 


LY Example 3 is another kind of question you will find in this booklet. 


Example 3 


What do you think? Tell how much you agree with these 


statements. 
Fill one circle for each line. 
Agree Agree Disagree _ Disagree 
a lot a little a little alot 

a) Watching movies is fun --------------- O O O O 

b) I like eating ice cream ----------------- O O O O 

c) Ido not like waking up early -------- CO O O O 

d) Ienjoy doing chores -------------------- O O O O 


LY Example 3 says, “What do you think? Tell how much you agree with these 
statements.” Letter (a) says, “Watching movies is fun.” You are given four choices 
for how much you agree with the statement: Agree a lot, Agree a little, Disagree a 
little, or Disagree a lot. 


LY Fill in the circle below your answer. For example, if you really agree a lot with that, 
fill in the first circle under “Agree a lot”. If you really disagree a lot, fill in the circle 
under “Disagree a lot”. 


Give students time to fill in their answers to all parts of the Example 3 question and make sure 
they understand how to answer this kind of question. Then continue reading the final directions: 


Read each question carefully, and pick the answer you think is best. 


Fill in the circle next to or under your answer. 


BBE 


If you decide to change your answer, draw an “X” through your first answer, as you 
see in the example. Then, fill in the circle next to or under your new answer. 


Ask for help if you do not understand something or are not sure how to answer. 


BB 


Are there any questions before we start? 
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Exhibit B-2. PIRLS Session Script—Continued 


If there are questions try to answer them the best you can. If there are no more questions, then 
record the current time in Cell (12a) of the Test Administration Form and proceed with the 
administration of the questionnaire. 


LQ Turn the page to the first question and begin answering this questionnaire. You will 
have 20 minutes to answer these questions. 


After 20 minutes are up, say: 


Ly Please stop working and raise your hand if you have finished answering the 
questions. 


If all of the students raise their hands, say: 


LY Thank you very much for participating in this study. Your work will help us to 
learn more about our students and schools. 


LY Please stay seated while I collect your questionnaires. 
If not all of the students raise their hands, allow for additional time and say: 


LY You will have more time to continue answering this questionnaire. If you have 
already finished all the questions, then you can use this time to review your answers. 
Once you have finished, please close your questionnaire and read quietly at your 
desk. 


Once all students have finished and have closed their questionnaires record the current time in Cell 
(12b) of the Test Administration Form. Then say: 


LY Thank you very much for participating in this study. Your work will help us to 
learn more about our students and schools. 


LO Please stay seated while I collect your questionnaires. 


Collect the questionnaires and keep them secure. Check against the Student Tracking Form to make 


sure that you have received all of them. 
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Exhibit B-3. ePIRLS Session Script 


Once all students are ready to begin, read the following: 


You have been chosen to take part in an important international study of how well 


ae) 


Ly 


children around the world can read online. 


First, I am going to explain the directions for using the computer to answer questions 
about what you have read, and then you will do two class projects that involve reading 
websites. Different students will be working on different projects. After your first 
project, there will be a short break, and then you will do your second project. 


If you still have any schoolbooks or papers on your desk, please put them away. Except 
for the computer you are using for ePIRLS, all electronic devices must be stored away. 


The reading projects you will be working on are already set up on your computer. 
Now, we are all going to work through the directions together so that you will know 
what to do. I will read the directions aloud while you follow along on your computer. 
We will go step by step, so please wait for me to tell you when to go on. 


First, please click on the “Password to Start” box. Type “1000” and click “Start” to begin. 


Is everybody looking at the screen that asks you to choose a boy or girl to represent you 
during the test? 


If yes, then continue. If not, help students get started. 


Please click on the boy or girl to represent you during the test. 


Non, click on the “NEXT” button to continue. 


Now you are introduced to the teacher who is going to help you through your class 
projects. Please read along on screen as I read the introduction to the teacher aloud. 


Hi. I am the teacher who is going to help you today. You will be reading some 
webpages and answering questions that I will ask you. You may find some parts easy 
and some parts difficult. 


LZ) Now, click on the “NEXT” button to continue. Everybody should be looking at the website 


called Polar Bear Facts. Please read along on screen as I read the teacher directions 
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Exhibit B-3. ePIRLS Session Script—Continued 


aloud. 


iin inter ishoreiqeoqranhic conicTeatiresinocsr bear i mae PIRLS 
Class Project 
Ms. Web: 
POLAR BEAR FACTS a You a two 
2 - projects. During each project, 


you will read information 
from several websites. 


Home Facts 


| ‘TheAmazingPolarBear *)-) .* * ire raras 


left called “Polar Bear Facts.” 
This website has two tabs, 
“Home” and “Facts.” If you 
click on a tab, it takes you to 
another webpage. 


Please click on the “Facts” 
tab now and read about the 
polar bear. 


n 
n 
a 
[=4 
1) 
le) 
a 
a 


= You will work on two projects. During each project, you will read information from 
several websites. 


= Look at the website on the left called “Polar Bear Facts.” This website has two tabs, 
“Home” and “Facts.” If you click on a tab, it takes you to another webpage. 


=, Please click on the “Facts” tab now and read about the polar bear. 


Read the following, while checking to see if students are on the web page shown in the next picture. 


LL) Everybody should be seeing three facts about polar bears on the webpage and some 
new instructions from the teacher. Please let’s continue reading along with the teacher. 


=, You will be asked some questions about each webpage you read. 


=, A red border will show you the question you are working on. 


(1) If it has not happened already, in a few seconds the first question will appear in the 
“Class Project” window. If you need help finding the “Facts” page please raise your 
hand. 
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Exhibit B-3. ePIRLS Session Script—Continued 


TIME LEFT | P 2011 


[Potr Gear Facts 7 Class Project 
POLAR BEAR FACTS 


Home 


| . ° 2 as . @ 
Here are some facts about polar bears: Waa 
For some questions you will 
eg) be asked to select your 
answer. Click on the circle 
next to your answer. 


If you are not sure about the 
answer, click on the answer 
that you think is best. 


un 
n 
uu 
4 
1U) 
2) 
a 
a 


1. 
What color is a polar bear’s fur? 


'e) Student 


() Now, please let’s continue reading along with the teacher. 


= For some questions you will be asked to select your answer. Click on the circle next to 
your answer. 


= Ifyou are not sure about the answer, click on the answer that you think is best. 


= Question 1 asks, “What color is a polar bear’s fur?” 
p 


(1) Now, please click on the answer that you think is best. 


After students have clicked on an answer, continue reading along with the teacher’s comments in the 


pop-up box. 


=, “White” is the correct answer. 


=, If you did not click on “White,” you can change your answer by clicking on another 
circle. 


=, When you have made your choice, click on the “SAVE” button to go on to the next 
question. 


Confirm that all students have clicked on the “SAVE” button, and then continue. 


=, For the next question, you will be asked to type your answer. 
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Exhibit B-3. ePIRLS Session Script—Continued 


= Do not worry if you are not sure how to spell a word. Do the best you can. 


=, Question 2 asks, “Where do polar bears live?” 


Now, type your answer in the box below the question. 


Confirm that all students are typing an answer in the open-response field, then read along with the 
teachers comments in the pop-up box. 


= When you finish typing, click on the “SAVE” button to go on to the next question. 


When the students have clicked the “SAVE” button, continue reading along with the teacher. 


=, Sometimes, I will ask you to click on a link to another website. Links are in blue and 
underlined. Now, click on the following link—A_Polar Bear’s Year. 


TIME LEFT PIRLS Online Ri 
http://animallearning org/polar-bears-year/ PIRLS 


" Class Project 


nw 
n 
tu 
« 
C) 
2) 
x 
a 


Ms. Webster 

For some webpages, you may 
need to scroll to see all of the} _ 
information. 7 


Click on the scroll bar next to 
the yellow arrow and drag it 
down to see the rest of this 
page. 


LL) Is everybody looking at a new web page, with four boxes about Fall, Winter, Spring, 
and Summer? 


If yes, continue reading along. 


=, For some webpages, you may need to scroll to see all of the information. 


=, Click on the scroll bar next to the yellow arrow and drag it down to see the rest of this 
page. 
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Exhibit B-3. ePIRLS Session Script—Continued 


Confirm that all students have scrolled, and then continue. 


ina 


Sometimes, a link is in the middle of a webpage. 


For example, the “dens” link in the “FALL” box has more information about polar 
bears’ dens. 


Now, click on the link “dens.” 


Is everybody reading about a den? 


If yes, continue. 


When you have read about dens, click on “CLOSE” to make the box go away. 


Confirm that all students have clicked on “CLOSE”, and then continue. 


i) 


i 


For the next question, you will need to use a drop-down menu. 


When the teacher asks you to use a drop-down menu, several possible answers to the 
question will appear or “drop down” when you click on the triangle. 


Click on the small black triangle next to “Choose what they do.” 


For this question, three answer choices drop down. Is everybody looking at the three 
choices? 


TIME LEFT IRLS eat , 
http://animallearning.org/polar-bears-year/ PIRLS 


- Class Project 


Because the website says “Polar bear cubs are born” in 


winter, the correct answer is “Give birth to cubs.” Ms. Webster 
. For the next question, you 
Click on “SAVE” to move to the next question. will need to use a drop-down 
menu. 


n 
a) 
pri] 
a 
iC) 
° 
4 
a 


Click on the small 

black triangle jx 

next to “Choose what they 
do.” 


3. 


What do female polar bears do in 
winter? 


‘ey Student 
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Exhibit B-3. ePIRLS Session Script—Continued 


If yes, continue. If no, help the students understand how to click on the triangle and make the drop- 


down menu appeat. 


=, Because the website says “Polar bear cubs are born” in winter, the correct answer is 
“Give birth to cubs.” 


= Click on “SAVE” to move to the next question. 


Confirm that all students have clicked on the “SAVE” button, and then continue. 


TIME LEFT PIRLS Onli 2016 
http://google.com?polar+bears+onticeberg PIRLS 


Class Project 


Google Polar bears on iceberg 
Stuffed Animals - Polar Bear Vhat do fem. 
www.stuffedsafari.com/polar-bears 
Find the perfect polar bear stuffed animal to chill out with. 


Polar Bears on Iceberg - Image Results 
www.arcticbear.org/polar-bear-on iceberg 
Pictures and videos of polar bears stranded on icebergs... 


Polar Bear Jokes 
www.animaljokes.com 
Q. What do polar bears eat for lunch? A.: ice-berg-ers! 


n 
n 
tu 
-4 
1] 
o 
4 
a 


Children's Stories - The Lost Polar Bear 


www.electrickids.com/stories 


A baby polar bear named Iceberg is lost in a big snow storm.. 


Ms. Webster 
Sometimes | will ask you to 
click on the best answer 


from the results of a Google 
search. 


Remember, click only on the 
link that gives your answer. 


4. 
Look at the Google search results, 
at left. 


'@) Student 


Click on the link that is most likely 
to show images of polar bears 


Google and the Google logo are registered trademarks of Google Inc., used with permission. stranded on icebergs. 


=, Sometimes I will ask you to click on the best answer from the results of a Google 
search. 


=, Remember, click only on the link that gives your answer. 


=, For Question 4, look at the Google search results, at left. 


=| Click on the link that is most likely to show images of polar bears stranded on icebergs. 


Confirm that students have clicked on one of the Google search results, then read along with the 


teachers comments in the pop-up box. 


= You should have clicked on “Polar Bears on Iceberg — Image Results.” It has been 
loaded for you. 
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Exhibit B-3. ePIRLS Session Script—Continued 


LO) Click on “X” to make the pop-up box go away. 


LL) Please note how the webpages you viewed appear as tabs at the top of the page. At this 
time, you can see tabs called “Polar Bear Facts,” “A Polar Bear’s Year,” and “Polar 
Bears on Iceberg.” 


http://articbear org/polar-bear-on-iceberg _ / PIRLS . 
Polar Bear Facis / A Polat Bear's Yea Class Project 
Bear.org 


Ms. Webster 
» 4 Sometimes | will ask you to 
<9) click on the best answer from 


the results of a Google 
search. 


Remember, click only on the 
link that gives your answer. 
4. 


Look at the Google search results, at 
left. 


'e) Student 


During summer, the weather turns warmer, the sea ice melts and breaks up. Polar (ie eT Lal ECE 
; c to show images of polar bears 
bears may become stranded on an ice floe. To escape, they may need to swim long stranded on icebergs. 
distances to get home. 


Ms. Webster 
| 1 Do your best to answer each 
< question. If you cannot 


answer a question, still click 
on SAVE to move on. 


wn 
n 
ty 
ia 
y 
fe) 
4 
a 


Confirm that students have located the tabs of the websites at the top of the page, and then 
continue. 


LL) If you need to refer to any of the previously viewed websites, you can go back by 
clicking on its tab. 


LL) Now, please let’s continue reading along with the teacher. 


=, Do your best to answer each question. If you cannot answer a question, still click on 
SAVE to move on. 


LL) Click on the “NEXT” button. The teacher now is going to explain very important 
information. So, please remember what the teacher says now. 


= After you click on SAVE, you will not be able to change that answer until you reach 
the end of the project. 


=, However, when you finish the last question, you will be able to review all your answers 
by scrolling back up through the questions. 
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Exhibit B-3. ePIRLS Session Script—Continued 


I have now finished explaining the directions. Please click “NEXT” and I will give you the 
password to start the first project. You will have 40 minutes to complete each project. 


Now, follow the script in Section 3.3. 


The First Class Project 


All students should now be looking at the “Password for First Project” screen, shown below. 


PIRLS 
Reading 
VANES 


Welcome to the ePIRLS 2016 
Online Reading Assessment 


You will have 40 minutes to work on your first project and then we will 
take a short break. Then you will work on a second project for another 40 
minutes. 


A clock at the top left of your screen will show how much time is left to 
work on the project. Below the clock there is progress bar. As you answer 

Ms.Webster each question, its box on the bar turns blue. If you decide not to answer a 
question, its box remains gray. You can go back to that question when you 
have finished. 


PROGRESS 


Now stop and wait for your test administrator to give you the password for 
your first project. 


Password for First Project 


) 


We will now begin work on the first project. Read each question carefully, if you are 
not sure about an answer to a question, give the answer you think is best and continue 
with the next question. 


You will have 40 minutes to work on your first project and then we will take a short 
break. Then you will work on a second project for another 40 minutes. 


A clock at the top left of your screen will show how much time is left to work on the 
project. Below the clock there is progress bar. As you answer each question, its box on 
the bar turns blue. If you decide not to answer a question, its box remains gray. You 
can go back to that question when you have finished. 


Make sure that everybody sees the clock in the upper left corner and the progress bar below it. 


When the clock in the top left corner shows that you have no time left, you will be 
automatically logged out of the project. If you finish before 40 minutes and have 
checked your work, you may log out yourself and read a book quietly if you have one 
easily accessible. 


Do you have any questions? 
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Exhibit B-3. ePIRLS Session Script—Continued 
When all problems, if any, are resolved, record the current time in Cell (8b) and also in Cell (9a) 


of the ePIRLS Test Administration Form and say: 


The password to begin the first project is “1835.” Please type the password in the box. 
Click “Start First Project” to start working. 


Make sure that all students are working on their first project. Remember that you are not allowed 
to help the students with navigating through the websites or answering the questions. 


About 5 minutes before the end of the session, say: 
You have about 5 minutes left before the break. 


If you have reached the end of the project, you may go back to any questions you have 
not answered. You can also check the answers of questions you have done. 


If you have finished and logged out, please wait for the others to finish. 


After the last 5 minutes have passed, say: 
Your time is up. 
At this time, record the current time in Cell (9b) of the ePIRLS Test Administration Form. 


£2) If you have not done it yet, please answer the question about how much you liked doing 
your project. For example, if you liked the project a lot, then click on the circle next to the 
smiling face and the words “I liked it a lot.” When ready, click “Save.” 


Make sure that all students have answered the question about how much they liked doing the 
project. If they need help answering the question, you are free to help them. 


When students have saved their answer to the question about how much they liked the first 
project, the Password for Second Project screen will reappear with a box for the password for 
the second task. 


We will now take a 5 minute break. Please leave the computer running on the 
“Password for Second Project” screen. 


Because the computers will be left with the ePJRLS Software running, do not leave the room 
unattended during the break. 

The Second Class Project 

After the break, record the current time in Cell (10a) of the ePIRLS Test Administration Form. 


Ask students to be seated at their computers. In order to make sure that each student is at the 
correct computer, say: 


Welcome back. Is everybody sitting at the same computer as before? 
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Exhibit B-3. ePIRLS Session Script—Continued 


If yes, continue. 
You will now have 40 minutes to work on your second project. 


After we complete the second project, I will ask you to answer the short 
ePIRLS Student Questionnaire about your experience with using computers. 


If you complete the second project before time is up and have reviewed your work, you 
may log out of the project and answer the question about how much you liked doing 
the second project. Please sit quietly or read a book at your computer station. 


Do you have any questions? 


When all problems, if any, are resolved, record the current time in Cell (10b) and also in 
Cell (11a) of the ePIRLS Test Administration Form. Then say: 


(4) We will now work on the second project. The password to begin the second project is 
“3972.” Please type in the password. Click “Start Second Project” to begin working on 
this project. You will have 40 minutes. 


About 5 minutes before the end of the session, say: 
You have about 5 minutes left to work on your second project. 


If you have reached the end of the project, you may go back to any questions you have 
not answered. You can also check the answers to questions you have done. 


If you have finished, please make sure to answer the question about how much you 
liked doing the project. When ready, click “Save.” Please wait quietly for others to 
finish. 


After the last 5 minutes have passed, say: 


Your time is up. Please stay seated so I can give you the password for the 
ePIRLS Student Questionnaire. 


At this time, record the current time in Cell (11b) of the ePIRLS Test Administration Form. 


Before giving students the password for the questionnaire, make sure that all students have 
answered the question about how much they liked doing the second project. If they need help, 
you are free to help them. 


After students have submitted their answer to the project liking question, the Password for 
Questionnaire screen will reappear with a box to enter the password to start the questionnaire. 
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Exhibit B-3. ePIRLS Session Script—Continued 
ePIRLS Student Questionnaire 


At this time, record the current time in Cell (12a) of the ePIRLS Test Administration Form. Then 
say: 
The password to begin the ePIRLS Student Questionnaire is “6317.” Please type in the 


password and click “Start Questionnaire.” The questionnaire should take you about five 
minutes to complete. 


Make sure that students complete the ePIRLS Student Questionnaire. If students need help 
answering these questions, you are free to help them. Please note that these questions have a 
“back” button. 
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Exhibit B-3. ePIRLS Session Script—Continued 


PIRLS 


Reading 


AUS 


ePIRLS 2016 Student Questionnaire 


@ About how much time do you spend using a computer each day? 


Less than 30 minutes 
30 minutes up to 1 hour 
From 1 hour up to 2 hours 


2 hours or more 


PIRLS 


Reading 


AUS 


ePIRLS 2016 Student Questionnaire 


€) About how much time do you spend each day finding and reading 
information on the Internet? 


Less than 30 minutes 


30 minutes up to 1 hour 


From 1 hour up to 2 hours 


2 hours or more 
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Exhibit B-3. ePIRLS Session Script—Continued 


PIRLS 


Reading 


PANS 


ePIRLS 2016 Student Questionnaire 


3 How much do you agree with these statements? 
Click on one button for each row. 


Agreea Agreea Disagree Disagree 
lot little a little alot 


1am good at using a computer 
lam good at typing 


It is easy for me to find information on the Internet 


€ BACK NEXT > 


PIRLS 


Reading 


PANTS 


ePIRLS 2016 Student Questionnaire 


@ Who mainly taught you the following things? 


Click on one button for each row. 


| mainly taught My a My | have never 
myself teachers sian friends learned this 


Using a computer 


Typing 


Finding information on 
the Internet 


Please raise your hand when you have finished answering the questionnaire and wait quietly. 


When students have submitted their answers to the questionnaire, the “Well done!” screen will 
appear. When all of the students have completed the questionnaire, say: 


Please do not turn off your computers or remove the USB sticks. Thank you very much 
for participating in this study. Your work will help us to learn more about our students 


and schools. 
At this time, record the current time in Cell (12b) of the ePIRLS Test Administration Form. 


You may now dismiss the students. 


Please answer questions 13 through 19 on the ePIRLS Test Administration Form and complete 
the Student Response Rate Form (see Section 3.3 in the PIRLS Test Administration Manual). 
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Exhibit B-4. Test Administration Form 


PIRLS 2016 


Test Administration Form 


Class ID: 


[1] 


[6] 


[7] 


Start time 


LT 


Test Administrator’s Position: 
[__]rms national center staff 


[__]reacher from school but not teacher of the selected class 


Cotte pnedeante §—§ [J 


Type of testing session:| _ Regular [| oake-up 


Date of testing:| _ DD: Day of testing [_}»m: Month of testing 


Year of testing:| _|2015 [_]2016 


End time BETS fe 


Administrative tasks (preparation of students, reading of 
instructions, distribution of achievement test booklets, etc.) 
a Le ek a sa 


LS LS < anise = 
Session for the Student Questionnaire (If the Student Questionnaire 
is administered on a different date than the test, enter the date) 
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Exhibit B-4. Test Administration Form—Continued 


13 
Were there any special circumstances or unusual events during the testing session (e.g., loud noises, 
students leaving or disturbing the testing session, or attempting to cheat, fire or smoke alarms, etc.)? 


No O Yes, please explain O 


14 
Did students compain about any particular problems with the testing (e.g., test too difficult, confusing, 
struggling with any particular item, etc.)? 
No O Yes, please explain O 


15 
Were there any problems with the testing materials (e.g., errors or omissions in the Student Tracking 
Forms, incorrect test booklet assignments, or insufficient booklets)? 
No O Yes, please explain O 


16 
Were there any students requiring special accommodations (e.g., students with visual or hearing 
impairment, dyslexia, etc.)? 
No O Yes, please specify the accommodation(s) provided O 


17 


Did a Quality Control Monitor observe the testing session? 


No O Yes, please explain O 
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Exhibit C-1. Student Questionnaire 


Identification Label 


PROGRESS IN INTERNATIONAL READING LITERACY STUDY 


Student 
Questionnaire 


Grade 4 


National Center for Education Statistics 
U.S. Department of Education 

550 12" St., SW, 4" floor 

Washington, DC 20202 
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Directions 


In this booklet, you will find questions about you and what you think. For each 
question, you should choose the answer you think is best. 


Let us take a few minutes to practice the kinds of questions you will answer in this 


booklet. 


Example 1 is one kind of question you will find in this booklet. 


Example 1 


Do you go to school? 


Fill one circle only. 


Yes -- © 
No--© 


Example 2 is another kind of question you will find in this booklet. 


Example 2 


How often do you do these things? 


Fill one circle for each line. 


Every day Once or Once or Never or 
or almost twicea twice a almost 
every day week month never 

a) I talk with my friends ------------------ Oo O O O 

b) Iplay sports -------------------+----------+ QO Co O O 

c) Iride a skateboard ---------------------- CG @ O O 
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Example 3 is another kind of question you will find in this booklet. 


Example 3 


What do you think? Tell how much you agree with these 


statements. 
Fill one circle for each line. 
Agree Agree Disagree Disagree 
a lot a little a little a lot 

a) Watching movies is fun --------------- O O O (e) 

b) [like eating ice cream ----------------- O O @) O 

c) Ido not like waking up early -------- O O O O 

d) I enjoy doing chores -------------------- '@ O O O 


@ Read each question carefully, and pick the answer you think is best. 
e Fill in the circle next to or under your answer. 


e If you decide to change your answer, draw an ».4 through your first answer, 
like this: R Then, fill in the circle next to or under your new answer. 


e Ask for help if you do not understand something or are not sure how to 
answer. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


APPENDIX C. QUESTIONNAIRES 


About you 


1 


A. Are you a girl or a boy? 
Fill one circle only. 
Girl -- © 


Boy -- QO 


B. Are you Hispanic or Latino? 
Fill one circle only. 
Yes, Iam Hispanic or Latino -- C2 


No, I am not Hispanic or Latino -- C) 


C. Which of the following best describes you? 
Fill one or more circles. 
White --E 
Black or African American --() 
Asian --() 
American Indian or Alaska Native --C 


Native Hawaiian or other 
Pacific Islander --(C 


jes | 
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When were you born? 


Fill the circles next to the month and year you were born. 


a) Month 


January -- 
February -- 
March -- 
April -- 
May -- 

June -- 

July -- 
August -- 
September -- 
October -- 
November -- 


December -- 
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OOO OO O30 OOOO.) 


b) Year 
2003 -- © 
2004 -- O 
2005 -- O 
2006 -- O 
2007 -- O 
2008 -- O 
2009 -- C 
Other -- O 


APPENDIX C. QUESTIONNAIRES 


A. How often do you speak English at home? 


Fill one circle only. 


I always speak 
English at home --() If always, please go to question 4 


I almost always speak 


English at home -- © 
I sometimes speak English 
and sometimes speak 
another language at home -- () 


I never speak English at home -- () 


If almost always, sometimes, or never, please go to question 3B 


B. What language do you speak at home (other than 
English)? 


Fill one circle only. 
Spanish -- CE 


Other -- © Please Specify 


cr | 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER’S GUIDE 


| C-8 | APPENDIX C. QUESTIONNAIRES 


A. Was your mother (or stepmother or female legal 
guardian) born in the United States? 


(United States” includes the 50 states, its territories, the District of 
Columbia, and U.S. military bases abroad.) 


ill one circle only. 


Yes --O 
No--© 
I don’t know -- C 


B. Was your father (or stepfather or male legal guardian) 
born in the United States? 


Fill one circle only. 


Yes --O 
No--© 
I don’t know -- C) 


C. Were you born in the United States? 


Fill one circle only. 


Yes --O 
No--O 
I don’t know -- C) 
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About how many books are there in your home? (Do not 
count magazines, newspapers, or your school books.) 


None or very few (O—10 books) -- 


Enough to fill one shelf (11-25 books) -- 


Enough to fill one bookcase 
(26-100 books) -- 


Enough to fill two bookcases 


(101-200 books) -- 


Enough to fill three or more bookcases 


(more than 200) -- 


ee Student Questionnaire 


Fill one circle only. 


O 


O 


‘ec 


O 


O 


This shows 10 books 


a 


This shows 25 books 


TTT 


This shows 100 books 
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Do you have any of these things at your home? 


Fill one circle for each line. 


Yes No 

a) A computer or tablet -------------------- O O 

b) Study desk/table for your use -------- O O 

c) Your own room -----------------+--++---+- © O 

d) Internet connection --------------------- O O 

e) Your own cell phone -------------------- O O 
f) A gaming system 

(e.g., PlayStation®, Wii®, XBox®) ---- OC O 

g) VCR, DVD, or Blu-ray player -------- O O 
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About how often are you absent from school? 
ill one circle only. 
Once a week -- C) 
Once every two weeks -- CO 
Once a month -- C) 


Never or almost never -- C2 


How often do you feel this way when you arrive at 


school? 

Fill one circle for each line. 

Every Almost 

day every day Sometimes Never 
a) I feel tired ---------------------------------- O GC O O 
b) I feel hungry ------------------------------- QO O O O 
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9 


How often do you eat breakfast on school days? 
Fill one circle only. 

Every day -- QO 

Most days -- C) 

Sometimes -- C) 


Never or almost never -- C2 


10 


How often do you use a computer or tablet in each of 
these places for schoolwork (including classroom tasks, 
homework, or studying outside of class)? 


Fill one circle for each line. 


Every day Onceor Once or Never or 
or almost twice a twice a almost 
every day week month never 

a) At home ------------------------------------- O QO O O 

b) At school ------------------------------------ QO O O O 

c) Some other place ------------------------ QO QO CG) (S) 
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APPENDIX C. QUESTIONNAIRES 


How much time do you spend using a computer or tablet 
to do these activities for your schoolwork on a normal 
school day? 


a) 


b) 


12 


Fill one circle for each line. 


30 minutes More than 


No time or less 30 minutes 
Finding and reading 
information -------------------------------- © O O 
Preparing reports and 
presentations ----------------------------- O O O 


How much time do you spend each day using a computer 
or tablet for any of the following activities? 


a) 
b) 
¢) 
d) 


Fill one circle for each line. 


Less than 30 minutes From 1hour 2 hours 

No time 30minutes uptolhour upto2hours or more 
Playing games --------- O O O @ O 
Watching videos ------ O O O O O 
Chatting ----------------- O O O O O 
Surfing the Internet - O O O O O 
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Your School 


13 


What do you think about your school? Tell how much 


you agree with these statements. 


ill one circle for each line. 


Agree 
a lot 


a) I like being in school -------------------- O 
b) I feel safe when I am at school ------ O 
c) I feel like I belong at this school ---- QO 


d) Teachers at my school are 
fair to me. ---------------------------------- O 


e) Iam proud to go to this school ------- O 
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Agree 
a little 


O 


Oe wae Ome) 


Disagree 
a little 


O 


Gy Oo «Oe 


Disagree 
a lot 


O 


ey HO 
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During this year, how often have other students from 
your school done any of the following things to you 
(including through texting or the Internet)? 


a) 


g) 


h) 


Made fun of me or called 


me names ------- 


Left me out of their 
games or activities ---------------------- 


Spread lies about me ------------------- 


Stole something from me -------------- 


Hit or hurt me 


(e.g., shoving, hitting, kicking) ------- 


Made me do things 
I didn’t want to do ----------------------- 


Shared embarrassing 
information about me ------------------ 


Threatened me 


Fill one circle for each line. 


At least Once or Afew 

once a twicea times a 

week month year Never 
O 1S O S 
O O O O 
O O O O 
O O O O 
O ‘* O O 
O @ O O 
O O O O 
O O © O 
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Lessons about reading 


15 


Think about the reading you do for school. How much 
do you agree with these statements about your reading 


lessons? 
Fill one circle for each line. 
Agree Agree Disagree Disagree 
a lot a little a little a lot 
a) I like what I read about in school --- QO CG) O O 
b) My teacher gives me interesting 
things to read ----------------------------- O O O O 
c) I know what my teacher 
expects me to do ------------------------- O O O O 
d) My teacher is easy to understand -- CO O O O 
e) Iam interested in what my 
teacher Says ------------------------------- O O O O 


f) My teacher encourages me to 
say what | think about what 
Thave read. -------------------------------- © 


g) My teacher lets me show what I 
have learned ------------------------------- 


h) My teacher does a variety of 
things to help us learn ----------------- O 


i) My teacher tells me how to do 
better when I make a mistake ------- QO 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


APPENDIX C. QUESTIONNAIRES | C-17 | 


Reading in school 


16 


In school, how often do these things happen? 


Fill one circle for each line. 


Every day Onceor Once or Never or 
or almost twice a twice a almost 
every day week month never 
a) Tread silently on my own ------------- O O OQ O 
b) LIread things that I choose myself -- O O O O 
c) My teacher asks us in class to 
talk about what we have read ------- O GC O O 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER’S GUIDE 


| C-18 | APPENDIX C. QUESTIONNAIRES 


Using the library 


17 


How often do you borrow books (including ebooks) 
from your school or local library? 


Fill one circle only. 
At least once a week -- CE) 
Once or twice a month -- C 
A few times a year -- C) 


Never or almost never -- C2) 
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Reading outside of school 


18 


How much time do you spend reading outside of school 
on a normal school day? 


Fill one circle only. 
Less than 30 minutes -- CE 
30 minutes up to 1 hour -- O 
From 1 hour up to 2 hours -- C) 


2 hours or more -- () 


19 


How often do you do these things outside of school? 


Fill one circle for each line. 


Every day Once or Once or Never or 
or almost twicea twice a almost 
every day week month never 
a) Lread for fun ----------------------------- O CG) O O 
b) I read to find out about things I 
want to learn ----------------------------- O O O O 
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What you think about reading 


20 


What do you think about reading? Tell how much you 
agree with each of these statements. 


Fill one circle for each line, 


Agree Agree Disagree Disagree 
a lot a little a little a lot 
a) [like talking about what I read 
with other people ------------------------ O O QO O 
b) I would be happy if someone gave 
me a book as a present ---------------- O O O ‘e 
ce) I think reading is boring -------------- O O O O 
d) I would like to have more 
time for reading -------------------------- C@) © O O 
e) lIenjoy reading --------------------------- O O QO O 
f) Ilearn a lot from reading ------------- O O © O 
g) I like to read things that 
make me think --------------------------- © O O @ 
h) I like it when a book helps 
me imagine other worlds -------------- (@ O O O 
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How well do you read? Tell how much you agree with 


each of these statements. 


a) I usually do well in reading ---------- 
b) Reading is easy for me ----------------- 


c) Ihave trouble reading 
stories with difficult words ----------- 


d) Reading is harder for me than 
for many of my classmates ----------- 


e) Reading is harder for me than 
any other subject. ------------------------ 


f) Iam just not good at reading -------- 


Fill one circle for each line. 


Agree Agree 
a lot a little 
O O 
O O 
O O 
O O 
O O 
O O 


Disagree 
a little 


O 


Onde a G3 


Disagree 
a lot 


O 


Ce GF 0). ES KD 
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Activities outside of school 


22 


The following questions ask about activities you do 
outside of school. 


Fill one circle for each line. 


Yes No 
a) Do you play on a sports team 
outside of school? ------------------------ O O 
b) Do you often play a musical 
instrument outside of school? -------- O @ 
ce) Are you studying something 
in a class outside of school? ---------- O 


d) Do you belong to a club outside of 
school (dike Girl Scouts, Cub Scouts, 
4-H, or Boys and Girls Club)? ------- O O 
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Thank you for filling out the questionnaire! 
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Exhibit C-2. School Questionnaire 


Identification Label 


PROGRESS IN INTERNATIONAL READING LITERACY STUDY 


School 
Questionnaire 


Grade 4 


National Center for Education Statistics 
U.S. Department of Education 

550 12" St., SW, 4" floor 

Washington, DC 20202 
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School Questionnaire 


Your school has agreed to participate in 

PIRLS 2016 (Progress in International Reading 
Literacy Study), an educational research project 
sponsored by the International Association for the 
Evaluation of Educational Achievement (IEA). PIRLS 
measures trends in student achievement in reading 
and studies differences in national education 
systems in more than 50 countries in order to help 
improve teaching and learning worldwide. 

This questionnaire is addressed to school 
principals and department heads who are asked to 
supply information about their schools. Since your 
school has been selected as part of a nationwide 
sample, your responses are very important in helping 
to describe primary/elementary education in the 
United States. 

Itis important that you answer each question 
carefully so that the information provided reflects 
the situation in your school as accurately as possible. 
Some of the questions will require that you look up 
school records, so you may wish to arrange for the 
assistance of another staff member to help provide 
this information. 


PIRLS 
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Since PIRLS is an international study and all 
countries are using the same questionnaire, you may 
find that some of the questions seem unusual or are 
not entirely relevant to you or schools in the United 
States. Nevertheless, it is important that you do your 
best to answer all of the questions so comparisons 
can be made across countries in the study. 

It is estimated that you will need approximately 
30 minutes to complete this questionnaire. We 
appreciate the time and effort that this takes and 
thank you for your cooperation and contribution. 

When you have completed the questionnaire, 
please place it in the accompanying envelope and 
return it to the PIRLS school coordinator. 

Your responses will be combined with those from 
other participants to produce summary statistics and 
reports. 

Thank you. 
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School Enrollment and Characteristics 


1 eee 4 SS ee 
What is the total enrollment of students in your 
school as of March 1, 2016? A. Approximately what percentage of students in 


your school have English as their native language? 
___ students Check one circle only. 
Write in the number. 
More than 90% --- C) 
76 to 90% --- C) 
51 to 75% --- C) 
26 to 50% --- C) 
25% or less --- C) 


2 


What is the total enrollment of fourth grade 
students in your school as of March 1, 2016? 


students 


Write in the number. 
B. Of the students currently enrolled in your school, 


what percentage have been identified as limited- 
English proficient (LEP)/English Language Learners 
(ELL)? 
3 — — — >|>~=_ Io“ — i“ _ |{*“~"“~=~E~““_=L_=___ == Check one circle only. 
Approximately what percentage of students in your 0% —-O 
school have the following backgrounds? 
1 to 5% --- C) 


Check one circle for each line. 


0%, --- 
as 6to 10% --- C) 
11 to 25% 11 to 25% -—- C) 
26 to 50% 26 to 50% --- C) 
More th 
50% 51t0 75% -— CO 
a) Come from economically 76 to 90% --- QO 


disadvantaged homes --------- O-—-O-—O-O 


b) Come from economically 


affluent homes------—-—------—-- O-O-O-O 


Over 90% --- C) 


a 
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A. How many people live in the city, town, or area 
where your school is located? 


Check one circle only. 
More than 500,000 people -- (_) 
100,001 to 500,000 people --- C) 
50,001 to 100,000 people --- C_) 
30,001 to 50,000 people -- C_) 
15,001 to 30,000 people --- C) 
3,001 to 15,000 people --- () 
3,000 people or fewer —-- C_) 


B. Which best describes the immediate area in which 
your school is located? 


Check one circle only. 
Urban—Densely populated -- (_) 


Suburban—On fringe or 
outskirts of urban area -- C_) 


Medium size city or large town —- C_) 
Small town or village -- C_) 
Remote rural --- C) 


C. Which best characterizes the average income level 
of the population in the school’s immediate area? 


Check one circle only. 


High -- OQ 


Medium --- ©) 
Low -- C) 
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6 _—————_—_—_—___ =A 
What type of school is this? 


Check one circle only. 
Regular public school --(_) 
Aregular public school with a magnet program --C_) 


A magnet school or school with a special program 

emphasis (e.g., Montessori, science/math school, 

performing arts school, talented/gifted school, 
foreign language immersion school, etc.) --C_) 


Special education: a school that primarily serves 
students with disabilities --) 


Alternative: a school designed to address the needs of 
students, typically at risk of educational failure, which 
cannot be met in regular schools --(_) 


Vocational --(_) 

Charter School --C_) 

Private (independent) --(_) 
Private (religiously affiliated) --(_) 


Other --C_) 


——EEE 


Does your school provide free meals for 
students? 


Check one circle for each line. 
Yes, for all students 
Yes, for some students 
No 


Around the first of October 2015, what percentage 
of students at this school were eligible to receive 
free or reduced-price lunches through the National 
School Lunch Program? 


percentage of students 
Write in the number. 
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Instructional Time 
For the fourth grade students in your school: A. Does your school provide a place where students 
; can work on their schoolwork before or after 
A. How many days per year is your school open for school? 
instruction? 
Check one circle only. 
__days Yes -—- 
Write in the number. O 
No-—-O© 
(If No, go to #11) 


B. What is the total instructional time, excluding 
breaks, in a typical day? 


If Yes, 


B. Ils someone available to assist them with their 


minutes 
schoolwork? 


Write in the number of minutes per day. 
Please convert the number of hours into minutes. 


Check one circle only. 


C. In one calendar week, how many days is the school 
open for instruction? 


Check one circle only. 


6 days -—- C) 


5 1/2days -- C) 
5days-- C) 
41/2 days -—- C) 


4days-- C) 
Other -- C) 


‘a 
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Resources and Technology 
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Does your school have a school library? 


Check one circle only. 


No--- C) 


(If No, go to #12) 


If Yes, 


A. Approximately how many books (print) with 
different titles does your school library have 
(excluding magazines and periodicals)? 


Check one circle only. 
250 ot fewer -- C_) 
251-500 -- C) 
501-2,000 --- C) 
2,001-5,000 -- C) 
5,001-10,000 --- C.) 
More than 10,000 --- C_) 


B. Approximately how many titles of magazines and 
other periodicals (print) does your school library 
have? 


Check one circle only. 
0-O 
1-5-—-O 
6-10-- CE 
11-30 -- C) 


31 or more --- C) 


C. Can students borrow print materials from the 
library to take home? 


Check one circle only. 
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Does the school provide access to digital books? 


Check one circle only. 


Yes --- C.) 
No--- C) 


13 ee 


How many computers (including tablets) does your 
school have for use by fourth grade students? 


computers 
Write in the number. 
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eee 


How much is your school’s capacity to provide 
instruction affected by a shortage or inadequacy 
of the following? 


Check one circle for each line. 


Not at all 
Alittle 
Some 
Alot 
A. General School Resources 
a) Instructional materials (e.g., 
textbooks) ---—----~---—-—-- O 


b) Supplies (e.g., papers, 
pencils, materials) ------------ C) 


c) School buildings and 


grounds ——------—----—-—---—- C) 


d) Heating/cooling and lighting 


systems, —-—-—-—--—-—-—-—-—--—- O 


e) Instructional space (e.g., 


Classrooms) ---------------—---- O 


f) Technologically competent 


staff -—-——-——-—-—-—---——- O 


g) Audio-visual resources 
for delivery of instruction 
(eg., interactive white 
boards, digital projectors) ---- ) 


oe A Se ED 
OU Oe Cae oa Ome. 
> COs 2 


O 
O 
O 


h) Computer technology for 
teaching and learning 
(e.g., computers or tablets 
for student use) ---------------- 


i) Resources for students with 


disabilities---------------------- CaF) 


B. Resources for Reading 
Instruction 


a) Teachers with a specialization 


in reading ---------------------- O=KO=—O—0) 


b} Computer software/ 
applications for reading 


instruction --—--------~—- COaO=0=—0 


c) Library resources (books, 


ebooks, magazines, etc.) --—- O=-—-O=-0=-O 


d) Instructional materials for 
reading (e.g., reading series, 


textbooks) ------------------—-- Oa OO 
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School Emphasis 
on Academic Success 
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How would you characterize each of the following 
within your school? 


Check one circle for each line. 
Very high 
High 
Medium 


Low 


a) Teachers’ understanding of 


the school’s curricular goals --- )) —C.) —Q)—O —©O 


b) Teachers’ degree of 
success in implementing 


the school’s curriculum -------- .) —() —Q —OQ—©O 


¢) Teachers’ expectations 
for student achievement ------ 


O 
O 
O 
O 


d) Teachers’ ability to 


inspire students —-—-—-------- ) —Q —O—O—©O 


e) Collaboration between 
school leadership (including 
master teachers) and 
teachers to plan instruction --- C_) 


f) Parental involvement 
in school activities ------------- O 


g) Parental commitment to 
ensure that students are 
ready to learn ------------------ 


O 


Oe © ee Oa © a 
Orme oe Om © a e 


h) Parental expectations for 
student achievement --—------ C.) 


i) Parental support for 
student achievement --—------ 


j) Students‘ desire to do 
well in school ------------------ 


k) Students’ ability to reach 
school’s academic goals ------- 


Se Oe @ ma © a 
CT AQ Ar: 


|) Students’ respect for 
classmates who excel 
academically ------------------- 


O 
O 
O 
O 


‘a 


School Discipline and Safety 
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To what degree is each of the following a problem 
among fourth grade students in your school? 
Check one circle for each line. 


Not a problem 
Minor problem 
Moderate problem 


Serious 
problem 


a) Arriving late at school --------- 


b) Absenteeism (i.e., 
unjustified absences) ---------- 


©) Classroom disturbance -------- 
d) Cheating ----------------------- 
e) Profanity ----------------------- 


f) Vandalism -—--———-—-------—- 


SEC2S2 07576 eee 
OD EC) Ge 
CkORCE CORO mee 
SEC LCR eres ©) 


Q)Thet 


h) Intimidation or verbal abuse 
among students (including 


texting, emailing, etc.) —---- )—Q—Q—©O 


i) Physical fights among 


students -—-----—-—---—--—----- )—O—-O—O 


j) Intimidation or verbal abuse 
of teachers or staff (including 


texting, emailing, etc.) —--- ) —Q)—Q —©O 


DD —E—E 


To what degree is each of the following a problem 
among teachers in your school? 


Check one circle for each line. 


Not a problem 
Minor problem 
Moderate problem 


Serious 
problem 


a) Arriving late or 
leaving early ------------------- 


b) Absenteeism ------------------- 


¢) Failure to complete 
the curriculum ----------------- 


Cy HO 
CP Ho 
2 ES) 
OF O79 


i’ 
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Students’ Literacy Readiness 
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About how many of the students in your school can 
do the following when they begin the first grade of 
primary/elementary school? 


Check one circle for each line. 
Less than 25% 
25-50% 
51-75% 


More than 
75% 


a) Recognize most of the 
letters of the alphabet --------- 


b) Read some words -------------- 


c) Read sentences ---------------- 


OOOO 
OH Oe) 


d) Read a story -------------------- 
e) Write letters of the alphabet - C_) 
f) Write some words ----------—- C_) 


OOO 2 
OO OS) 
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APPENDIX C. QUESTIONNAIRES 


Reading in Your School 


_————— a 
At which grade do the following reading skills 
and strategies first receive a major emphasis in 
instruction in your school? 

Check one circle for each line. 


First grade or earlier 


Second grade 
Third grade 
Fourth 
grade 
Notin 
these 
grades 


a) Knowing letters of the 
alphabet ----------------------- QO 

b) Knowing letter-sound 
relationships ------------------- 


Ro): 15 


OP oe OL I) 
De PR RD, SD OE CD » 5) 
Gr DG: 
CS GF Dn > GE 


©) Reading words ----------------- 


d) Reading isolated sentences --- O 


e) Reading connected text ------- C) 


f) Locating information within 
the text. ---------------------—-- 


g) Identifying the main idea 
of a text ------------------------ 


h) Explaining or supporting 
understanding of a text ------- 


i) Comparing a text with 
personal experience ----------- 


j) Comparing different texts ----- O 
k) Making predictions about 


what will happen next in 
a text ——--—------------------—-- 


O 
O 
@) 
O 


|}, Making generalizations and 
drawing inferences based 
ona text ----------—-------—- 


m) Describing the style or 
structure of a text. ------------- 


O 
O 
O 
O 


n) Determining the author's 
perspective or intention ------- O 
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21 


22 


23 


Principal Experience and 
Education 


By the end of this school year, how many years will 
you have been a principal altogether? 


years 
Please round to the nearest whole number. 


By the end of this school year, how many years will 
you have been a principal at this school? 


years 
Please round to the nearest whole number. 


What is the highest level of formal education you 
have completed? 


Check one circle only. 


Did not complete a 4-year college or 
university degree (i.e., Bachelor's degree) ---_) 


Completed a 4-year college or university 
degree (i.e., Bachelor's degree) ---_) 


Completed a Master's degree, 

postgraduate certificate program 

(e.g., teaching), or professional degree 
(e.g., law, medicine, dentistry) --) 


Completed a doctorate (Ph.D. or Ed.D.) -—-C) 


Do you hold any of the following professional 
qualifications in educational leadership? 


Check one circle for each line. 


Yes 
No 


a) Certificate or license -—----~----------------—-- CJ) — CE) 


b) A Master's degree, postgraduate 
certificate program (e.g., teaching), or 
professional degree (e.g., law, medicine, 
dentistry) --——-—————-—-—--—-—-—-—---—-—-—---- C ) = C) 
O=-O 


¢) Adoctorate (Ph.D. or Ed.D.)----------------------- 


a 


APPENDIX C. QUESTIONNAIRES | C-33 | 


Thank you for the thought, time, and effort you have 
put into completing this questionnaire. 
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Exhibit C-3. Teacher Questionnaire 


Identification Label 


PROGRESS IN INTERNATIONAL READING LITERACY STUDY 


Teacher Questionnaire 


Grade 4 


National Center for Education Statistics 
U.S. Department of Education 

550 12" St., SW, 4" floor 

Washington, DC 20202 
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Teacher Questionnaire 


Your school has agreed to participate in 

PIRLS 2016 (Progress in International Reading 
Literacy Study), an educational research project 
sponsored by the International Association for 

the Evaluation of Educational Achievement (IEA). 
PIRLS measures trends in student achievement in 
reading and studies differences in national education 
systems in more than 50 countries in order to help 
improve teaching and learning worldwide. 

This questionnaire is addressed to teachers of 
fourth grade students, and seeks information about 
teachers’ academic and professional backgrounds, 
classroom resources, instructional practices, and 
attitudes toward teaching. Since your class has 
been selected as part of a nationwide sample, your 
responses are very important in helping to describe 
primary/elementary education in the United States. 

Some of the questions in the questionnaire refer 
to the “PIRLS class’ or “this class.’ This is the class 
that is identified on the front of this booklet, and 
which will be tested as part of PIRLS in your school. 

If you teach some but not all of the students in the 
PIRLS class, please think only of the students that you 
teach when answering these class-specific questions. 
It is important that you answer each question 
carefully so that the information that you provide 


reflects your situation as accurately as possible. 


PIRLS 
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Since PIRLS is an international study and all 
countries are using the same questionnaire, you may 
find that some of the questions seem unusual or are 
not entirely relevant to you or schools in the United 
States. Nevertheless, it is important that you do your 
best to answer all of the questions so comparisons 
can be made across countries in the study. 

It is estimated that you will need approximately 
35 minutes to complete this questionnaire. We 
appreciate the time and effort that this takes and 
thank you for your cooperation and contribution. 

When you have completed the questionnaire, 
please place it in the accompanying envelope and 
return it to the PIRLS school coordinator. 

Your responses will be combined with those from 
other participants to produce summary statistics and 
reports. 

Thank you. 
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About You 


1 


5 SS—S—S—S—S[[__===—=—==—=_===_—=s 
A. During your college or university education, what 


By the end of this school year, how many years will 


you have been teaching altogether? 


years 
Please round to the nearest whole number. 


2 


Are you female or male? 


Check one circle only. 


How old are you? 


Check one circle only. 
Under 25 -- C_) 
25-29 -- C) 
30-39 -- OC) 
40-49 -- C) 
50-59 -- C) 


60 or more --- O 


) EH 


What is the highest level of formal education you have 
completed? 


Check one circle only. 
Did not complete high school --- C_) 
Completed high school --- C_) 


Completed a 2-year college 
or university degree 
(i.e. Associate's degree) —-O- 


Completed a 4-year college or 
university degree 
(i.e., Bachelor's degree) ---C_) 


Completed a Master's degree, 

postgraduate certificate program 

(e.g., teaching), or professional 
degree (e.g., law, medicine, dentistry) --- C) 


Completed a doctorate 
(Ph.D. or Ed.D) --C) 
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was your major or main area(s) of study? 


Check one circle for each line. 


Yes 
No 
a) Education—Primary/Elementary -----~------- C) = C.) 
b) Education—Secondary ------------------------- C) — C) 


dj fndlish eae Ce) 
dOthaiia ae CC) 


. As part of your formal education and/or training, 


to what extent did you study the following areas? 


Check one circle for each line. 


Not at all 


Overview or introduction 
to topic 


It was an area 
of emphasis 


AE bha eC) 
b) Literature --------------------- C) 


c) Pedagogy/teaching reading - C_) 
d) Educational psychology ------- C) 


e) Remedial reading ----------—- C_) 
f) Reading theory --------------- (_) 
g) Special education -—--—---—-—- C) 


h) Second language learning ---- QO 


i) Assessment methods 
in reading ---------------------- 


OG Reser eCnOr ere es Ee. 
OO: GOD 2: OG 


j) Early childhood education --—- C) 


a 


F———— 


In the past two years, how many hours in total have 
you spent in formal professional development (e.g., 
workshops, seminars, lesson studies, etc.) that dealt 
directly with reading or teaching reading (e.g., 
reading theory, instructional methods)? 


Check one circle only. 
None --- C_) 
Less than 6 hours -- (_) 
6-15 hours -- C) 
16-35 hours -- C.) 
More than 35 hours --- C.) 
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School Emphasis on 
Academic Success 


7 


How would you characterize each of the following 
within your school? 


Check one circle for each line. 
Very high 
High 
Medium 


Low 


a) Teachers’ understanding of 


the school’s curricular goals -- ) —) —Q —QO—©O 


b) Teachers’ degree of 
success in implementing 


the school’s curriculum ---—-—-- O-O-O-O-O 


c) Teachers’ expectations 
for student achievement ------ O 


@) 
S) 
O 
O 


d) Teachers’ ability to 


inspire students ---------------- e666 @ 8 


e} Collaboration between school 
leadership (including master 
teachers) and teachers to 


plan instruction ---------------- O-O-0O-0-O 


f) Parental involvement 
in school activities ------------- 


O 
O 
O 


g) Parental commitment to 
ensure that students are 


ready to learn -—-------------—- O=-—-O=-0=-0=-0 
h) Parental expectations for 

student achievement ---—-—- O-—-O-O-O-O 
i) Parental support for 

student achievement ----—-—-- O-—-O-O-O-O 
j) Students’ desire to do 

well in school -------------—- O-—-O-O-O-O 
k) Students’ ability to reach 

school's academic goals -—-—-- O—-O-—-0=-0=0 
|) Students’ respect for 

classmates who excel 

academically --------------—-—-- O-—-O-—-O0=—-O0=-0O 
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School Environment 


Thinking about your current school, indicate the 
extent to which you agree or disagree with each of 
the following statements. 


Check one circle for each line. 


Agree a lot 
Agreea little 
Disagree a little 
Disagree 
alot 
a} This school is located in 
a safe neighborhood -------—-- O 


b) | feel safe at this school -----—-- O 


c) This school's security policies 
and practices are sufficient ---- 


d) The students behave in an 


Os SO? 4s SY SOI Or 
Je Cn tO» SO, RO A) 
Ce Hs RS Me Os OI 


orderly manner ---------------- © 
e) The students are respectful 

of the teachers --------------- O 
f) The students respect 

school property ---------------- O 
g) This school has clear rules 

about student conduct -------- O 


h) This school's rules are 
enforced ina fair and 
consistent manner --------—-—-- O 


O 
O 
O 
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About Being a Teacher 
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How often do you have the following types of 
interactions with other teachers? 


Check one circle for each line. 


Very often 
Often 
Sometimes 


Never or 
almost never 


a) Share what | have 
learned about my 


teaching experiences -----—--— O-O-O-O 


b) Observe another classroom 


to learn more about teaching - ) =) =) =O 


c) Work together to 
improve how to teach a 


particular topic --—-—--------— O-O-O-O 


d) Work with teachers from 
other schools on the 


curriculum ------------------—- O O O O 


e) Work with teachers from 
other grades to ensure 


continuity in learning --------- O=O=0=06 


10 


How often do you feel the following way 
about being a teacher? 


Check one circle for each line. 


Very often 
Often 
Sometimes 
Never or 
almost never 
a) |am content with my 
profession asa teacher ------— O-—-O=-0=-0O 
b) | find my work full of 
meaning and purpose -------- O-O-O-O 
c) Lam enthusiastic 
about my job -—-—-—-—-—-—----— O-0C-O-O 
d) My work inspires me ------—--—- O-—-O-O-O 
O=-0O-0 


e) | am proud of the work Ido -- C_) 


‘a 
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About Teaching Reading to the PIRLS Class 


11 


A. How many students are in this class? 


students 
Write in the number. 


B. How many of the students in #11A are in 
fourth grade? 


fourth grade students 
Write in the number. 


DV) 


How many fourth grade students experience 
difficulties understanding spoken English? 


students in this class 
Write in the number. 


BS ——"————xz 


A. How many students need remedial instruction 
in reading? 


fourth grade students in this class 
Write in the number. 


B. How many of the students in #13A receive 
remedial instruction in reading? 


students 
Write in the number. 


Se eee] 


How many students in the class are advanced 
readers? 


fourth grade students in this class 
Write in the number. 


—P>o——E—EEElEiElEEEE 


In your view, to what extent do the following limit 
how you teach this class? 
Check one circle for each line. 


Not atall 
Some 
Alot 


a) Students lacking prerequisite 
knowledge or skills ------------ 


b) Students suffering from 
lack of basic nutrition ---------- 


©) Students suffering from 
not enough sleep -------------- 


d) Students absent from class ---- 


e) Disruptive students ------------ 


OO Or” 
OG) rH" 


2) 
O 
O 
O 
O 
O 


f} Uninterested students --------- 


g) Students with mental, 
emotional, or psychological 
impairment -------------------- 


O 
‘2 


h) Lack of support for using 
information technology ----—-- C) 


O 
O 
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In a typical week, how much time do you spend on When you have reading instruction and/or do 
English language instruction and/or activities with reading activities, how often do you organize 
the students? students in the following ways? 

Include instruction or activities in reading, writing, Check one circle for each line. 


speaking, literature, and other language skills. 
Always or almost always 


7 Often 
minutes per week : 
to i ; Sometimes 
Write in the number of minutes per week. 
Please convert the number of hours into minutes. Never 


a) | teach reading as a 
whole-class activity --------- C.) 
b) I create same-ability groups -- ) 
Regardless of whether or not you have formally 


scheduled time for reading instruction, in a typical ¢) I create mixed-ability groups - C) 
ca about how besa do you = ik d) Nuseindividwalized 

reading instruction and/or activities with the instruction for reading ------— O 
students? 


e) Students work independently 


Include things you do across curriculum areas and on anassigned plan or goal --- () 


during formally scheduled time for reading instruction. 


©). AY ay) 
Oe, © me Oe OS 
Or. 2G) a) 


minutes per week 
Write in the number of minutes per week. 
Please convert the number of hours into minutes. 


‘a 
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About Teaching Reading to the PIRLS Class 


When you have reading instruction and/or do 
reading activities with the students, how often 
do you have the students read the following 
types of text (in print or digitally)? 


Check one circle for each line. 
Every day or almost every day 
Once or twice a week 


Once or twice a 
month 


Never or 
almost never 


A. Literary Reading Materials 
a) Short stories (e.g., fables, fairy 
tales, action stories, science 

fiction, detective stories) ------ 


b) Longer fiction books with 
chapters ------------------------ 


¢) Plays. ---------------—— 


Os 
O 
O 
O 


B. Informational Reading 
Materials 


a) Nonfiction subject area 


books or textbooks ----—------ )—() = OQ =O 


b) Longer nonfiction books with 


chapters -—---—---—-----—--- ) —O—O—O 


©) Nonfiction articles that 
describe and explain about 
things, people, events, or 
how things work 
(e.g., newspaper articles, 


brochures) ---—----—-—--—------- Q—O—O—O 


20 EEE 


When you have reading instruction and/or do 
reading activities with the students, how often 
do you do the following? 


Check one circle for each line. 
Every day or almost every day 
Once or twice a week 


Once or twice a 
month 


Never or 
almost never 


a) Read aloud to students -------- O 
b) Ask students to read aloud --- O 


c) Ask students to read silently 
on their own ------------------- 


d) Teach students strategies for 
decoding sounds and words -- 


e) Teach students new 
vocabulary systematically ----- O 


f) Teach students how to 
summarize the main ideas ---- _) 


g) Teach or model skimming 
or scanning strategies --------- 


Os Re Ee. GD 
Oust Ole Om O an © © 5G 
Oe Ole Oe Oe © ae Oe 
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2) EE 


How often do you do the following in teaching 
reading to this class? 


Check one circle for each line. 


Every or almost every lesson 
About half the lessons 


Some lessons 
Never 


a) Provide reading materials 
that match the 


students’ interests-----------—-- O-—-O-O-O 


b) Provide materials that are 
appropriate for the reading 


levels of individual students—-.)) = =) — © 


c) Link new content to 


students’ prior knowledge ----- _) O Ss O 


d) Encourage students to 
develop their understandings 


of the text ===> C—O =—( j=) 


e) Encourage student discussions 


of texts --------------------—---- ete ee 


f) Encourage students to 
challenge the opinion 


expressed in the text -----—-—-- O-—-O-O=-O 


g) Use multiple perspectives 
(among students and texts) 


to enrich understanding ---—-- O-O-O0O-O 


h) Give students time to read 
books of their own choosing -- _) 


© 
O 
QO 


i) Give individualized feedback 


to each student ---------------- O-—-O-—-O-O 
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How often do you ask the students to do the 
following things to help develop reading 
comprehension skills or strategies? 


Check one circle for each line. 


Every day or almost every day 
Once or twice a week 


Once or twice a 
month 


Never or 
almost never 


a} Locate information within 


the text -—-—---——----—------ )— OD —-O—O 


b) Identify the main ideas 


of what they have read ~------ ) —() —Q —Q) 


c) Explain or support their 
understanding of what they 


have ead. <-=——————=------ J —$ O—O—-O 


d) Compare what they have 
read with experiences they 


have had -----------------------  ) —Q)—Q—O 


e) Compare what they have 
read with other things they 


have read. --—-——————-—-=------- J—$ O—O—-O 


f) Make predictions about 
what will happen next in the 


text they are reading ~-------- )—C)—O —O 


g) Make generalizations and 
draw inferences based on 


what they have read ----------- =) =Q —© 


h) Describe the style or 
structure of the text they 


have read --——--—-—----——--- )—Q —O—O 


i) Determine the author's 


perspective or intention -------.) —Q) —Q) —© 


a 


 ——————————————————————— 


After students have read something, how often do 
you ask them to do the following? 


Check one circle for each line. 


Every day or almost every day 
Once or twice a week 


Once or twice a 
month 


Never or 
almost never 


a) Write something about or 
in response to what they 


have read -—---—-—-—-—--—----- Q—Q—O—O 


b) Answer oral questions 
about or orally summarize 


what they have read ---------- 1) =() =Q) =) 


c) Talk with each other about 
what they have read ----------- 


O 
O 
O 


d) Take a written quiz or test 


about what they have read ---- C_) O O O 
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A. Do the students in this class have computers 
(including tablets) available to use for their 
reading lessons? 


Check one circle only. 


Yes -- C) 
No--- C) 


(If No, go to #25) 


If Yes, 


B. What access do the students have to computers? 


Check one circle for each line. 


Yes 
No 


a) Each student has a computer ------------------ (.) — C) 


b) The class has computers that students 
can shate --—-—-—-—-—--—-—--------------------- QD = OE 


c) The school has computers that the class 
can use sometimes —-------------------—-—--- () = () 


C. How often do you do the following computer 
activities during reading lessons? 


Check one circle for each line. 


Every day or almost every day 
Once or twice a week 


Once or twice a 
month 


Never or 
almost 
never 


a) Ask students to read 


digital texts —-—-——---—--- ]) —O—Q—O 


b) Teach students strategies 


for reading digital texts ------ .) —C) —Q) —O 


©) Teach students to be 
critical when reading on 


the Internet --------—----------- ) —])—Q) —O 


d) Ask students to look up 
information (e.g., facts, 


definitions, etc.) ---—-—-—----- 0D —-O—-O—O 


e) Ask students to research 
a particular topic 


or problem ------------------- ) —O—O—O 


f) Ask students to write 


stories or other texts --—-----) —) —QO—O 
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ee 


A. Do you have a library or reading corner in your 
classroom? 


Check one circle only. 


Yes --- C) 
No--- C) 


(If No, go to #26 ) 
If Yes, 
B. About how many books are in your classroom 
library? 


Check one circle only. 
0-25 --- C) 
26-50--- CE) 
51-100 --- C.) 
More than 100 --- C) 


C. About how many magazines with different titles 
are in your classroom library? 


Check one circle only. 
0—O 
1-2---C) 
3-5-- CO) 
More than 5 --- C_) 


D. How often do you give the students in your 
class time to use the classroom library or 
reading corner? 


Check one circle only. 
Every day or almost every day --- _) 
Once or twice a week --- (_) 
Once or twice a month --- C_) 
Never or almost never --- C_) 


E. Can the students borrow books from the classroom 
library or reading corner to take home? 


Check one circle only. 


Yes --- C) 
No--- C) 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER'S GUIDE 


a 


How often do you take or send the students toa 
library other than your classroom library? 


Check one circle only. 
At least once or twice a week —-- (_) 
Once or twice a month -- C_) 
Afew times a year --- C_) 


Never or almost never -- C_) 


10 


Reading Homework 


27 


How often do you assign reading as part of 
homework (for any subject)? 


Check one circle only. 


| do not assign reading 
for homework --- C) 


(Go to #30) 
Less than once a week --- (_) 
1 or 2times a week --- C) 
3 or 4times a week -- (_) 


Every day --- C) 


238 EE 


In general, how much time do you expect students 
to spend on homework involving reading (for any 
subject) each time you assign it? 


Check one circle only. 
15 minutes ot less -- C_) 
16-30 minutes --- C) 
31-60 minutes --- C) 
More than 60 minutes --- _) 


29 Ee 


How often do you do the following with the reading 
homework assignments for this class? 


Check one circle for each line. 


Always or almost always 
Sometimes 


Never or 
almost never 


O=-O 
b) Discuss the homework 


in class ret a) 'S Co 


©) Monitor whether or not the 
homework was completed ----.)—=() =) 


a) Correct assignments and give 
feedback to students ---------- 
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30 SSE eee ey 
Are the following resources available to you to work 
with students who have difficulty with reading? 

Check one circle for each line. 


Always 
Sometimes 


Never 


a) Aspecialized professional 
(e.g., reading specialist, 
speech therapist) -------------- 


GaQ=0 
b) Ateacher-aide -—-————----- ) —)—© 
Q=O0=0 


c) Anadult/parent volunteer ---- 


em ry 


What do you usually do if a student begins to fall 
behind in reading? 


Check one circle for each line. 


Yes 
No 

a) | have the student work with a specialized 

professional (e.g., reading specialist, 

speech therapist) ---------—-—-—-—------—-—----—-- O=-O0O 
b) | wait to see if performance improves with 

Maturation -—------—---=-=---—-=-—-——nnnnnnnnnn— O=-O 
c) | spend more time working on reading 

individually with that student ---—-------------- O-O 
d) | ask the parents to help the student 

with reading ------------------—-----------n=n-—=- O=-O 
e) | recommend that the student be 

enrolled ina special reading program ----—---—- O-O 
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Assessing Reading 


32 


How much emphasis do you place on the 
following sources to monitor students’ 
progress in reading? 


Check one circle for each line. 
Major emphasis 
Some emphasis 


Little or no 
emphasis 


a) Assessment of students’ 


ongoing work ------—----------- O-O-O 


b) Classroom tests (for 
example, teacher-made 


or textbook tests) -----------—-- O-O-O 


©) State or district 


achievement tests ----------—-- O=-O=-O 
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Thank you for the thought, time, and effort you have 
put into completing this questionnaire. 


U.S. PIRLS AND EPIRLS 2016 TECHNICAL REPORT AND USER’S GUIDE 


| C-48 | APPENDIX C. QUESTIONNAIRES 


Exhibit C-4. ePIRLS Student Questionnaire 


PIRLS 


Reading 


PATS 


ePIRLS 2016 Student Survey 


@ About how much time do you spend using a computer each day? 


Less than 30 minutes 
From 30 minutes to 1 hour 
From 1 hour to 2 hours 


2 hours or more 


PIRLS 


Reading 


PANTS 


ePIRLS 2016 Student Survey 


@ About how much time do you spend each day finding and reading 
information on the Internet? 


Less than 30 minutes 
From 30 minutes to 1 hour 


From 1 hour to 2 hours 


2 hours or more 
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PIRLS 


Reading 


PATS 


ePIRLS 2016 Student Survey 


3) How much do you agree with these statements? 
Click on one button for each row. 


Agreea Agreea Disagree Disagree 
lot little a little alot 


| am good at using a computer 


| am good at typing 


It is easy for me to find information on the 
Internet 


PIRLS 


Reading 


PATS 


ePIRLS 2016 Student Survey 


© Who mainly taught you the following things? 


Click on one button for each row. 


| mainly taught My My | have never 


My famil 
myself teachers "friends —_ learned this 


Using a computer 


Typing 


Finding information on the 
Internet 
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PIRLS 


Reading 


PAUTS 


ePIRLS 2016 Student Survey 


Click the BACK button to review your answers. 
Click the SUBMIT button to finish the survey. 


€ BACK SUBMIT 
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Exhibit C-5. Curriculum Questionnaire 
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Double click on the image to open the full document. 


© IEA, 2016 


PIRLS 2016 
Curriculum Questionnaire 


PIRLS § PIRLS 
its WI 


TIMSS & PIRLS 
International Study Center 
Tir hed af alata, Mena Ont 
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National Adaptations to PIRLS Questions 


2016 International Version 


2016 U.S. Adapted Version 


students can work on their schoolwork 
before or after school? 
Check one circle only. 
1. Yes 
2. No 
(If No, go to #9) 


students can work on their schoolwork 
before or after school? 
Check one circle only. 
1. Yes 
2. No 
(If No, go to #11) 


International National 
Questionnaire |Item Number | Item Item Number _ {Item Recoding Instructions 
School Questionnaire Adaptations 
School ScQ-01 What is the total enrollment of students in 1 What is the total enrollment of students 
your school as of <first day of month PIRLS in your school as of March 1, 2016? 
testing begins, 2016>? Write in the number. 
Write in the number. students 
Students 
School ScQ-02 What is the total enrollment of <fourth 2 What is the total enrollment of fourth 
grade> students in your school as of <first grade students in your school as of 
day of month PIRLS testing begins, 2016>? March 1, 2016? 
Write in the number. Write in the number. 
_s Students _ Students 
School ScQ-04 Approximately what percentage of students 4A Approximately what percentage of 
in your school have <language of test> as students in your school have English as 
their native language? their native language? 
Check one circle only. Check one circle only. 
1. More than 90% 1. More than 90% 
2. 76 to 90% 2. 76 to 90% 
3. 51 to 75% 3. 51 to 75% 
4. 26 to 50% 4. 26 to 50% 
5. 25% or less 5. 25% or less 
School ScQ-07 For the <fourth grade> students in your 9 For the fourth grade students in your 
school: school: 
School ScQ-08A Does your school provide a place where 10A Does your school provide a place where 
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National Adaptations to PIRLS Questions 


2016 International Version 


2016 U.S. Adapted Version 


International National 

Questionnaire |ltem Number | Item Item Number — {Item Recoding Instructions 

School ScQ-09 Does your school have a school library? 11 Does your school have a school library? 
Check one circle only. Check one circle only. 

1. Yes 1. Yes 
2. No 2. No 
(If No, go to #10) (If No, go to #12) 

School ScQ-11 How many computers (including tablets) 13 How many computers (including tablets) 
does your school have for use by <fourth does your school have for use by fourth 
grade> students? grade students? 

Write in the number. Write in the number. 
computers computers 

School ScQ-14 To what degree is each of the following a 16 To what degree is each of the following 
problem among <fourth grade> students in a problem among fourth grade students 
your school? in your school? 

Check one circle for each line. Check one circle for each line. 
1. Not a problem 1. Not a problem 
2. Minor problem 2. Minor problem 
3. Moderate problem 3. Moderate problem 
4. Serious problem 4. Serious problem 
School ScQ-16 About how many of the students in your 18 About how many of the students in your 


school can do the following when they 
begin the <first grade> of 
primary/elementary school? 

Check one circle for each line. 

1. Less than 25% 

2. 25-50% 

3. 51-75% 

4. More than 75% 


school can do the following when they 
begin the first grade of 
primary/elementary school? 

Check one circle for each line. 

1. Less than 25% 

2. 25-50% 

3. 51-75% 

4. More than 75% 
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National Adaptations to PIRLS Questions 


2016 International Version 


2016 U.S. Adapted Version 


your school, what percentage have been 
identified as limited-English proficient 
(LEP)/English Language Learners 
(ELL)? 

Check one circle only 

1.0% 

2.1t0 5% 

. 6 to 10% 

. 11 to 25% 

. 26 to 50% 

. 51 to 75% 

. 76 to 90% 

. Over 90% 


ANDNAAW 


International National 
Questionnaire |ltem Number | Item Item Number — {Item Recoding Instructions 
School ScQ-17 At which grade do the following reading 19 At which grade do the following reading 
skills and strategies first receive a major skills and strategies first receive a major 
emphasis in instruction in your school? emphasis in instruction in your school? 
Check one circle for each line. Check one circle for each line. 
1. <First grade> or earlier 1. First grade or earlier 
2. <Second grade> 2. Second grade 
3. <Third grade> 3. Third grade 
4. <Fourth grade> 4. Fourth grade 
5. Not in these grades 5. Not in these grades 
School ScQ-20 What is the highest level of formal 22 What is the highest level of formal 
education you have completed? education you have completed? 
Check one circle only. Check one circle only. 
1. Did not complete <Bachelor’s or equivalent 1. Did not complete a 4-year college or 
level—ISCED Level 6> university degree (i.e., Bachelor's degree) 
2. <Bachelor’s or equivalent level—ISCED 2. Completed a 4-year college or university 
Level 6> degree (i.e., Bachelor's degree) 
3. <Master’s or equivalent level—ISCED Level 3. Completed a Master’s degree, 
7> postgraduate certificate program (e.g., 
4. <Doctor or equivalent level—ISCED Level teaching), or professional degree (e.g., law, 
8> medicine, dentistry) 
4. Completed a doctorate (Ph.D. or Ed.D.) 
School ScQ-21b <Master’s or equivalent level—ISCED Level 7> 23b A Master’s degree, postgraduate certificate 
program (e.g., teaching), or professional 
degree (e.g., law, medicine, dentistry) 
School ScQ-21c <Doctor or equivalent level—ISCED Level 8> 23c A doctorate (Ph.D. or Ed.D.) 
School U.S. Specific Questions Added 
School 4B Of the students currently enrolled in 
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National Adaptations to PIRLS Questions 


2016 International Version 


2016 U.S. Adapted Version 


International 
Item Number 


Item 


National 
Item Number 


Item 


Recoding Instructions 


School 


5C 


Which best characterizes the average 
income level of the population in the 
school’s immediate area? 

Check one circle only. 

1. High 

2. Medium 

3. Low 


School 


What type of school is this? 

Check one circle only. 

1. Regular public school 

2. A regular public school with a magnet 
program 

3. A magnet school or school with a special 
program emphasis (e.g., Montessori, 
science/math school, performing arts 
school, talented/gifted school, foreign 
language immersion school, etc.) 

4. Special education: a school that primarily 
serves students with disabilities 

5. Alternative: a school designed to 
address the needs of students, typically at 
risk of educational failure, which cannot be 
met in regular schools 

6. Vocational 

7. Charter School 

8. Private (independent) 

9. Private (religiously affiliated) 

10. Other 


School 


Around the first of October 2015, what 
percentage of students at this school 
were eligible to receive free or reduced- 
price lunches through the National 
School Lunch Program? 

Write in the number. 

____ percentage of students 
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National Adaptations to PIRLS Questions 


2016 International Version 


2016 U.S. Adapted Version 


experience difficulties understanding 
spoken <language of test>? 
Write in the number. 

students in this class 


experience difficulties understanding 
spoken English? 
Write in the number. 


students in this class 


International National 
Questionnaire |ltem Number | Item Item Number — {Item Recoding Instructions 
Teacher Questionnaire Adaptations 
Teacher TQG-04 What is the highest level of formal 4 What is the highest level of formal Int -> Nat 
education you have completed? education you have completed? 1->1 
Check one circle only. Check one circle only. 2->2 
1. Did not complete <Upper secondary 1. Did not complete high school 4->3 
education—ISCED Level 3> 2. Completed high school 5 ->4 
2. <Upper secondary education—ISCED Level 3. Completed a 2-year college or university |6 -> 5 
3> degree (i.e., Associate’s degree) 7->6 
3. <Post-secondary, non-tertiary education— 4. Completed a 4-year college or university | International Category 3 
ISCED Level 4> degree (i.e., Bachelor's degree) (ISCED Level 4) is not 
4. <Short-cycle tertiary education—ISCED 5. Completed a Master's degree, administered 
Level 5> postgraduate certificate program (e.g., 
5. <Bachelor’s or equivalent level—ISCED teaching), or professional degree (e.g., law, 
Level 6> medicine, dentistry) 
6. <Master’s or equivalent level—ISCED Level 6. Completed a doctorate (Ph.D. or Ed.D) 
7> 
7. <Doctor or equivalent level—ISCED Level 
8> 
Teacher TQG-05A During your <post-secondary> education, 5A During your college or university 
what was your major or main area(s) of education, what was your major or main 
study? area(s) of study? 
Check one circle for each line. Check one circle for each line. 
1. Yes 1. Yes 
2. No 2. No 
TQG-05Ac <language of test> 5Ac English 
TQG-05Ba <language of test> 5Ba English 
Teacher TQR-01B How many of the students in #R1A are in 11B How many of the students in #11A are in 
<fourth grade>? fourth grade? 
Write in the number. Write in the number. 
<fourth grade> students fourth grade students 
Teacher TQR-02 How many <fourth grade> students 12 How many fourth grade students 
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National Adaptations to PIRLS Questions 


2016 International Version 


2016 U.S. Adapted Version 


International National 
Questionnaire |ltem Number | Item Item Number — {Item Recoding Instructions 
Teacher TQR-03A How many students need <remedial> 13A How many students need remedial 
instruction in reading? instruction in reading? 
Write in the number. Write in the number. 
<fourth grade> students in fourth grade students in 
this class this class 
Teacher TQR-03B How many of the students in #R3A receive 13B How many of the students in #13A 
<remedial> instruction in reading? receive remedial instruction in reading? 
Write in the number. Write in the number. 
students students 
Teacher TQR-04 How many students in the class are 14 How many students in the class are 
advanced readers? advanced readers? 
Write in the number. Write in the number. 
<fourth grade> students in this fourth grade students in 
class this class 
Teacher TQR-06 In a typical week, how much time do you 16 In a typical week, how much time do you 
spend on <language of test> language spend on English language instruction 
instruction and/or activities with the and/or activities with the students? 
students? Include instruction or activities in 
Include instruction or activities in reading, reading, writing, speaking, literature, 
writing, speaking, literature, and other and other language skills. 
language skills. Write in the number of minutes per 
Write in the number of minutes per week. week. Please convert the number of 
Please convert the number of hours into hours into minutes. 
minutes. minutes per week 
minutes per week 
Teacher TQR-22c National or regional achievement tests 32C State or district achievement tests 
Student Questionnaire Adaptations 
Student SQG-03 How often do you speak <language of test> 3A How often do you speak English at 


at home? 

Fill one circle only. 

1. | always speak <language of test> at 
home 

2. | almost always speak <language of test> 
at home 

3. | sometimes speak <language of test> 
and sometimes speak another language at 
home 

4. | never speak <language of test> at home 


home? 

Fill one circle only. 

1. lalways speak English at home 

2. | almost always speak English at 
home 

3. | sometimes speak English and 
sometimes speak another language at 
home 

4. | never speak English at home 


If always, please go to question 4 


If almost always, Sometimes, or Never, 
please go to question 3B 
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Questionnaire 


National Adaptations to PIRLS Questions 


2016 International Version 


2016 U.S. Adapted Version 


International 
Item Number 


Item 


National 
Item Number 


Item 


Recoding Instructions 


Student 


SQG-05 


Do you have any of these things at your 
home? 

Fill one circle for each line. 

1. Yes 

2. No 


6 


Do you have any of these things at your 
home? 

Fill one circle for each line. 

1. Yes 

2. No 


<country-specific indicator of wealth> 


6E 


Your own cell phone 


Student 


<country-specific indicator of wealth> 


6F 


A gaming system (e.g., PlayStation®, Wii®, 
XBox®) 


Student 


<country-specific indicator of wealth> 


6G 


VCR, DVD, or Blu-ray player 


Student 


U.S. Specific Questions Added 


Student 


1B 


Are you Hispanic or Latino? 
Fill one circle only. 

1. Yes, | am Hispanic or Latino 

2. No, | am not Hispanic or Latino 


Student 


1c 


Which of the following best describes 
you? 

Fill one or more circles. 

1. White 

2. Black or African American 

3. Asian 

4. American Indian or Alaska Native 

5. Native Hawaiian or other Pacific Islander 


Student 


Student 


3B 


4A 


What language do you speak at home 
(other than English)? 

Fill one circle only. 

1. Spanish 

2. Other, Please Specify 


Was your mother (or stepmother or 
female legal guardian) born in the 
United States? 


(“United States” includes the 50 states, its 
territories, the District of Columbia, and 
U.S. military bases abroad. 


Fill one circle only. 
1. Yes 

2. No 

3. | don’t know 
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National Adaptations to PIRLS Questions 


2016 International Version 2016 U.S. Adapted Version 
International National 
Questionnaire |ltem Number | Item Item Number — {Item Recoding Instructions 
Student 4B Was your father (or stepfather or male 
legal guardian) born in the United 
States? 


Fill one circle only. 
1. Yes 

2. No 

3. | don’t know 


Student 4c Were you born in the United States? 


Fill one circle only. 
1. Yes 

2. No 

3. | don’t know 


Student 22 The following questions ask about 
activities you do outside of school. 
Fill one circle for each line. 


1. Yes 
2. No 

Student 22A Do you play on a sports team outside of 
school? 

Student 22B Do you often play a musical instrument 


outside of school? 


Student 22C Are you studying something in a class 
outside of school? 


Student 22D Do you belong to a club outside of school 
(like Girl Scouts, Cub Scouts, 4-H, or Boys 
and Girls Club)? 
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